Lost in Transcription

From raw audio to real-world use in a low-resource language

Marija Stojcheva, MSc  &  Hristijan Gajdov  at  PyData

VEZILKA, FINKI / UKIM, EuroHPC JU, Grant No. 101263128

Partners: VEZILKA, AI Factories, EuroHPC Joint Undertaking, co-funded by the European Union, FINKI, UKIM, Ministry of Digital Transformation

One lecture clip, four transcripts

A held-out FINKI lecture clip (11 s). Press play, then compare the four transcripts.

Off-the-shelf models rarely fail with silence. They drift into a neighbouring language: Bulgarian words, Russian spelling.

Where we are going

Part 1

Why Macedonian is hard

What makes a language “low-resource” for speech recognition?

Low-resource is about labels, not speakers

Large multilingual models know about Macedonian. They have not heard enough of it.

Off the shelf, dialects break everything

A model that looks solved on read speech can still lose one word in three on real regional speech.
Part 2

How modern ASR works

Two architectures we build on

Whisper: an encoder that listens, a decoder that writes

The 30-second window is a design constraint we keep meeting: in segmentation, in filtering, and in latency.

Moonshine: built for short clips on small devices

A model that has never heard Macedonian still has a vocabulary that can spell it. That made full fine-tuning feasible.
Part 3

The data

Where 3,456 hours of Macedonian speech came from, and what we did to it

vezilka-ai/macedonian-speech

Two sources hold 98.6% of the hours. Every dialect together is under 35 hours.

From long recordings to training pairs

Most of the work was not training. It was turning hours of loosely matching audio and text into clean 30-second pairs.

Doniraj: a platform for donating your voice

Every new voice and every new region helps the next model hear Macedonian the way it is really spoken.

A benchmark humans listened to, clip by clip

The drafting model is one of the models we evaluate. That is fine only because a human corrected every draft, on audio it never trained on.
Part 4

Our models

Adapt a big model cheaply, then teach small ones with it

Our open models on Hugging Face

Approach A: LoRA on Whisper

Replay is the cheap insurance against catastrophic forgetting: keep a slice of what the model already knew in every batch.

Approach B: a teacher labels, small students learn

Distillation buys scale, and the student inherits the teacher's mistakes along with its knowledge.

Results on the FINKI lecture benchmark

A 27 M-parameter model trained on our data beats every hosted system we tried.

Accuracy, size, speed: pick two

Parameter count does not predict latency on a GPU. Measure on the hardware you will deploy to.
Part 6

Beyond transcription

Dialects, who spoke when, and asking questions of hours of audio

From Marija's master's thesis: “Toward Unified Speech Recognition and Understanding in Low-Resource and Dialect-Rich Languages”.

LoRA closes most of the dialect gap

The same recipe transfers: on Swiss German dialects it cuts WER by up to 50% against the best zero-shot baseline.

Who spoke when: diarization across sessions

Speaker-attributed transcripts turn a wall of text into a conversation you can analyse.

Ask the recordings: graph-based RAG

Chunk by speaker turn, with context. Mixing speakers in one chunk drops faithfulness from 0.84 to 0.50.
Part 7, use case

Дојава · Dojava

A voice fault line: a call comes in, a field team goes out

From a voice message to a dispatched team

A caller to the electricity or district-heating fault line hears: “After the tone, say your name, describe the problem, then your address and town.” Dojava does the rest.

Speech recognition is one stage of six. Its errors flow into every stage after it, which is why 4% WER matters.

What we learned

Low-resource is a data problem first. Open data and open models are how it gets solved.

Limits and next steps

Honest limits

  • The benchmark is 5 lectures and 4 speakers: one domain, no confidence intervals
  • WER punishes valid dialect spellings; orthography and recognition errors mix
  • Macedonian dialect corpora are small and narrow, so dialect WERs may not carry over to other speakers and settings
  • Diarization was assessed qualitatively; RAG judges are weak in Macedonian

Next

  • More dialect speech: regions we barely cover (Tikvesh: 97 clips) and regions we have no recordings of yet
  • A gold-standard benchmark for ASR, diarization and RAG beyond lectures
  • Semi-supervised training on unlabelled recordings

Use it, and donate your voice

huggingface.co/vezilka-ai

  • vezilka-whisper-large-v3-turbo-mk-lora
  • vezilka-moonshine-base-mk
  • vezilka-moonshine-tiny-mk-lr2e-4
  • vezilka-moonshine-tiny-mk-lr5e-4
  • datasets/macedonian-speech

Every dialect clip we have is a few hundred voices. Read a few sentences in yours: it takes five minutes and directly improves the next model.

QR code: huggingface.co/vezilka-ai
Models and data
QR code: doniraj.vezilka.ai
doniraj.vezilka.ai

Thank you · Ви благодариме

Teaching Macedonian AI takes data and care. Questions?

Supported by the Ministry of Education and Science of North Macedonia (“Utilizing AI and National Large Language Models to Advance Macedonian Language Capabilities”) and co-funded by EC/EuroHPC JU, the Ministry of Digital Transformation and FINKI through the VEZILKA project, Grant Agreement No. 101263128. Regional recordings: SYNDI project, Ministry of Culture (2021). Contact: info@vezilka.ai

Partners: VEZILKA, AI Factories, EuroHPC Joint Undertaking, co-funded by the European Union, FINKI, UKIM, Ministry of Digital Transformation