Building a Speech Datasetfor a Low-Resource Language

Voice AI for languages like Nepali is limited less by models than by data. There is far less publicly available, well-prepared conversational speech than there is for English, so teams building voice products end up preparing their own.

We recently built a pipeline that turns very long, raw, multi-speaker conversational recordings, used with permission, into a clean, segmented speech dataset. It runs on an ordinary laptop with almost no free disk space, survives interruptions, and scales to more audio by simply running it again.

The interesting parts were not the algorithms. They were the surprises in the data, two bugs that validators did not catch, and a handful of design rules we would now apply to any data pipeline.

The first step wrote nothing except a few short sample clips. We probed every file, decoded each one end to end to check integrity, compared the stereo channels, measured silence, and listened to clips spread across the recording.

That inspection changed the plan. Two files that looked distinct were byte-for-byte identical. The stereo audio was really mono, with identical channels, so splitting channels would never separate speakers. And roughly 30 percent of each recording was dead air, mostly in long blocks of several minutes.

After removing dead air and keeping only detected speech with a little padding, a 24-hour recording yielded about 14 hours of usable segments. That ratio, a bit under 60 percent, is now how we estimate collection targets: plan raw hours at well above the usable hours you need.

The earlier attempt at this preprocessing had run in a hosted notebook and kept failing. The cause was simple: it loaded entire recordings, each around 24 hours long, into memory.

The rebuilt pipeline never holds a whole file. One ffmpeg process streams each recording into ten-minute chunks at the sample rate the downstream models expect, and every later step works chunk by chunk, so memory use stays flat no matter how long the source is.

That step surfaced a subtle bug in how ffmpeg writes FLAC chunks, which we have written up separately on the Blogs page. The short version: the files played fine, passed the reference validator after a header fix, and still could not be read by the library the next stage used.

To find the speech inside each chunk we used Silero VAD, a small, widely used voice activity detection model. Instead of installing a full deep learning framework, which the machine had no disk space for, we ran its published ONNX model with onnxruntime and ported its standard post-processing.

Porting meant testing. The self-test compares our streaming loop against the reference implementation's loop and requires bit-identical output, alongside audio-level checks on synthetic speech, tones and noise. When the suite passed on the first try, we deliberately injected two classic streaming bugs, a missing context window and a dropped recurrent state, to make sure it would fail. The audio-level checks alone missed both. The bit-identical comparison caught them.

That habit, testing the tests, is cheap and has saved us more than once. A suite that has never been seen to fail has not been shown to work.

Make every step resumable. Each stage records its state in a small SQLite manifest, marks work as done only when its results are written in one transaction, and wipes and redoes anything left half-finished. Interrupting a run and resuming produced byte-identical output to an uninterrupted run, and we tested exactly that.

Store offsets, not copies. The segmentation stage writes no audio at all, only start and end times inside existing chunks. On a nearly full disk, that saved gigabytes and made re-segmenting with new settings free.

Flag, do not delete. Quality filters set a keep flag and a reason instead of removing data, so changing a threshold later needs no reprocessing, and rejected material is still there to audit.

Guard the disk. Every stage checks free space continuously and aborts cleanly, with no half-written state, before the machine runs out.

The original plan included a music and advertisement filter and a language identification step. Before building them, we exported short random samples of kept segments, rejected segments, and the unusual long stretches of unbroken speech, and listened.

The long stretches turned out to be crosstalk, people talking over each other, plus a single snippet of singing. Many short segments were backchannels, the brief acknowledgements listeners make while someone else talks. For conversational voice models, both are exactly the behaviour you want the model to learn, so we kept them and dropped the filtering step entirely.

To keep short segments useful, each one records the gap since the previous segment, and consecutive segments are grouped into conversation threads. Training code can stitch threads back into long, natural exchanges instead of seeing isolated fragments.

The final dataset stores audio, not model-specific tokens. Tokenizing for a particular speech codec is fast on a GPU and can be redone for any future model, while re-running the preprocessing is not.

The validation split is made of whole chunks rather than random segments, so neighbouring audio from the same conversation never leaks between training and evaluation.

Permission was scoped carefully. The recording owner consented, but other voices in the recordings did not consent individually, so the dataset is private and documented as not for redistribution. Provenance, every processing step and every parameter are recorded in the dataset card.

We built the pipeline with an AI coding agent, one stage at a time, with an engineer reviewing the results and the test report before the next stage started. Each stage had to prove itself: a full run, an idempotent re-run, an interrupted run that resumed cleanly, and a disk-guard abort.

Those reviews are where both FLAC problems and a miscount in the long-segment analysis surfaced. Agentic tooling makes building fast, and the review discipline is what makes the output trustworthy.

Agentic AI engineering and scalable systems are the two things Darvi Labs focuses on, and data pipelines like this one sit right between them. We work with verifiable steps, tests that are seen to fail, and clear records of where every piece of data came from.

If you are preparing speech or other training data for an AI product, especially in a language the big datasets overlook, get in touch through our Contact page. More of our engineering writing is on the Blogs page.