AI Data Engineeringfor Low-Resource Languages

A note first: Darvi Labs is a software engineering company in Kathmandu that does this kind of work, so we have an interest in the topic. The questions below apply whoever you hire.

If you are building voice or language AI for a language like Nepali, the bottleneck is usually not the model. Strong open models exist. What does not exist, or exists only in small or poorly documented amounts, is clean, consented, well-structured data in your language and domain.

Good AI data engineering turns raw material you are allowed to use into a dataset you can trust: documented provenance, reproducible processing, quality checks a human has actually listened to or read, and a format the next stage can use without rework.

For English, teams can often start from large public datasets and focus on fine-tuning. For Nepali, the public data is thinner, narrower in domain and less conversational. Customer-service speech, regional accents, and the natural mixing of Nepali with English are especially underrepresented.

That means most of the value is created upstream: finding usable sources, securing permission, cleaning raw recordings or text, and structuring it so models learn the behaviour you actually need, such as turn-taking in conversation or domain vocabulary in support calls.

It also means mistakes compound. A pipeline that silently drops or corrupts a fraction of a small dataset hurts far more than the same bug on a huge one.

Provenance and consent are recorded, not assumed. Every file can be traced to its source and the permission that covers it, and the scope of that permission is respected, including whether the data may be shared or only used privately.

Processing is reproducible. Every step is scripted, versioned and parameterised, models used in processing are pinned by checksum, and running the pipeline again produces the same output.

Pipelines are resumable and incremental. Adding new data should mean running the same pipeline again, processing only what is new, not starting over.

Humans review samples. Automated metrics catch some problems. Listening to or reading random samples of what was kept and what was rejected catches the rest, and often changes the plan.

Raw-but-clean beats prematurely processed. Keeping cleaned source data, rather than only model-specific features, lets you switch models later without redoing the expensive work.

How do you document provenance and consent, and what do you do with data whose permission scope is unclear?

Can you show us a pipeline that resumes cleanly after an interruption, and the test that proves it?

How do you validate files, and do you test the read path your training code will use, not just a format validator?

How much of the output does a person review before you call a dataset done, and how are rejected items recorded?

Where will our data be stored and processed, who can access it, and is anything sent to third-party services?

Vague answers about where the data comes from, or confidence that publicly visible content is automatically free to train on.

One-off notebooks instead of scripted, versioned pipelines. They work once and cannot be audited or rerun.

Deleting data during filtering with no record of what was removed or why.

No sampling or human review, only aggregate metrics.

Datasets delivered in a single model-specific format with the cleaned source audio or text discarded.

We build data pipelines and AI systems from Kathmandu, and we work with Nepali language data first-hand. You can read a technical account of one of our recent speech data pipelines, and a bug it uncovered, on our Blogs page.

If you are planning a voice or language AI product for a market the big datasets overlook, get in touch through our Contact page. We are happy to start with a small, measurable pilot.