Lost in Transcription

From raw audio to real-world use in a low-resource language

Marija Stojcheva, MSc  &  Hristijan Gajdov  at  PyData

VEZILKA, FINKI / UKIM, EuroHPC JU, Grant No. 101263128

Partners: VEZILKA, AI Factories, EuroHPC Joint Undertaking, co-funded by the European Union, FINKI, UKIM, Ministry of Digital Transformation

One lecture clip, four transcripts

A held-out FINKI lecture clip (11 s). Press play, then compare the four transcripts.

Off-the-shelf models rarely fail with silence. They drift into a neighbouring language: Bulgarian words, Russian spelling.

Where we are going

Part 1

Why Macedonian is hard

What makes a language “low-resource” for speech recognition?

Low-resource is about labels, not speakers

Large multilingual models know about Macedonian. They have not heard enough of it.

One recording, two correct transcripts

Ohrid regional speech (SYNDI), 6 s. Same audio, two layers in vezilka-ai/macedonian-speech.

“Correct” depends on the target. We keep both layers on the same audio, so the choice stays explicit.

Off the shelf, dialects break everything

A model that looks solved on read speech can still lose one word in three on real regional speech.
Part 2

How modern ASR works

Two architectures, one way of scoring them

Whisper: an encoder that listens, a decoder that writes

The 30-second window is a design constraint we keep meeting: in segmentation, in filtering, and in latency.

Moonshine: built for short clips on small devices

A model that has never heard Macedonian still has a vocabulary that can spell it. That made full fine-tuning feasible.

How we score: word error rate

Every number in this talk is normalised WER on audio the models never trained on.
Part 3

The data

Where 3,456 hours of Macedonian speech came from, and what we did to it

vezilka-ai/macedonian-speech

Two sources hold 98.6% of the hours. Every dialect together is under 35 hours.

From long recordings to training pairs

Most of the work was not training. It was turning hours of loosely matching audio and text into clean 30-second pairs.

Doniraj: a platform for donating your voice

Every new voice and every new region helps the next model hear Macedonian the way it is really spoken.

Preprocessing: small rules, big consequences

Small preprocessing rules decide what the model learns: clean audio, consistent text, honest splits.

A benchmark humans listened to, clip by clip

The drafting model is one of the models we evaluate. That is fine only because a human corrected every draft, on audio it never trained on.
Part 4

Our models

Adapt a big model cheaply, then teach small ones with it

Our open models on Hugging Face

Approach A: LoRA on Whisper

Replay is the cheap insurance against catastrophic forgetting: keep a slice of what the model already knew in every batch.

Approach B: a teacher labels, small students learn

Distillation buys scale, and the student inherits the teacher's mistakes along with its knowledge.

Engineering: 3 million clips, zero local copies

One recipe, two architectures: check what each tokenizer actually emits before you train.

Results on the FINKI lecture benchmark

A 27 M-parameter model trained on our data beats every hosted system we tried.

Hear it for yourself

Three held-out benchmark clips. Pick a tab, press play, then read what each system heard.

Accuracy, size, speed: pick two

Parameter count does not predict latency on a GPU. Measure on the hardware you will deploy to.

What still breaks

A fix that sounds right still has to win on the benchmark. Measure before you ship.
Part 6

Beyond transcription

Dialects, who spoke when, and asking questions of hours of audio

From Marija's master's thesis: “Toward Unified Speech Recognition and Understanding in Low-Resource and Dialect-Rich Languages”.

LoRA closes most of the dialect gap

The same recipe transfers: on Swiss German dialects it cuts WER by up to 50% against the best zero-shot baseline.

Who spoke when: diarization across sessions

Speaker-attributed transcripts turn a wall of text into a conversation you can analyse.

Ask the recordings: graph-based RAG

Chunk by speaker turn, with context. Mixing speakers in one chunk drops faithfulness from 0.84 to 0.50.
Part 7, use case

Дојава · Dojava

A voice fault line: a call comes in, a field team goes out

From a voice message to a dispatched team

A caller to the electricity or district-heating fault line hears: “After the tone, say your name, describe the problem, then your address and town.” Dojava does the rest.

Speech recognition is one stage of six. Its errors flow into every stage after it, which is why 4% WER matters.

A call, live: every stage streams in

Recording of the Dojava console processing the unclear-report sample call from Kochani
  1. Speech to text lands first: waveform, timestamped segments, 267 ms.
  2. Correction shows its edits as a diff.
  3. Category “Other”, confidence 0.76.
  4. Address “Сончева долина” is not in OpenStreetMap for Kochani.
  5. So the ticket goes to a dispatcher, with the reasons on top.

Recorded from the running console: the “unclear report, unknown street, Kochani” sample call, 2.0 s end to end.

Confident call, part 1: hear it, understand it

Speech-to-text stage of the heating call
Speech to text, 1.0 s on a Whisper server: “…since yesterday the radiators are completely cold, the heating does not work in the whole building, and we have a small baby and elderly neighbours.”
Category stage: no heating, high urgency
Category: no heating, high urgency, confidence 1.00, scope: the building.
Fields stage: name, town, street and problem
Fields pulled out by rules: name, town, street and the problem in the caller's words.

The LLM corrector made no edits: the transcript was already right. Personal data was masked first: the classifier saw [ИМЕ], not the caller's name.

Confident call, part 2: find it, send a crew

Address stage: verified OpenStreetMap match
Address fuzzy-matched against an offline OpenStreetMap gazetteer: verified at 0.99, with the near-misses listed.
Team stage: Heating 1 assigned
Team: the nearest free district-heating crew: Heating 1 (Парно 1), 3.5 km away, arriving in 15 minutes, with the next option listed.
4.2 s from the end of the call to an assigned crew, with no human in between. Critical calls are dispatched at once too, but flagged: “confirm immediately”.

Unsure call: a human decides

Ticket header with the review reasons
Correction diff with two punctuation edits
The LLM fixed punctuation (2 edits, shown as a diff), but no model can verify a street that is not on the map.
Review queue with capable teams sorted by distance
Review queue: capable, free teams first, sorted by distance. One click assigns.

The whole flow in one view

  • New report: play a sample call, upload audio or record one; the stages stream in live.
  • Map: the report location and the assigned crew's route to it.
  • Reports: the queue; “Needs review” tickets wait for a dispatcher.
  • Teams: who is free, who is in the field, and where.
The Ohrid call in the console: pipeline on the left, crew route on the map
The Ohrid call in the console: the pipeline on the left, the crew's route on the map.
Every screen in this part is recorded from the running system, replaying the calls exactly as it processed them.

What we learned

Low-resource is a data problem first. Open data and open models are how it gets solved.

Limits and next steps

Honest limits

  • The benchmark is 5 lectures and 4 speakers: one domain, no confidence intervals
  • WER punishes valid dialect spellings; orthography and recognition errors mix
  • Macedonian dialect corpora are small and narrow, so dialect WERs may not carry over to other speakers and settings
  • Diarization was assessed qualitatively; RAG judges are weak in Macedonian

Next

  • More dialect speech: regions we barely cover (Tikvesh: 97 clips) and regions we have no recordings of yet
  • A gold-standard benchmark for ASR, diarization and RAG beyond lectures
  • Semi-supervised training on unlabelled recordings

Use it, and donate your voice

huggingface.co/vezilka-ai

  • vezilka-whisper-large-v3-turbo-mk-lora
  • vezilka-moonshine-base-mk
  • vezilka-moonshine-tiny-mk-lr2e-4
  • vezilka-moonshine-tiny-mk-lr5e-4
  • datasets/macedonian-speech

Every dialect clip we have is a few hundred voices. Read a few sentences in yours: it takes five minutes and directly improves the next model.

QR code: huggingface.co/vezilka-ai
Models and data
QR code: doniraj.vezilka.ai
doniraj.vezilka.ai

Thank you · Ви благодариме

Teaching Macedonian AI takes data and care. Questions?

Supported by the Ministry of Education and Science of North Macedonia (“Utilizing AI and National Large Language Models to Advance Macedonian Language Capabilities”) and co-funded by EC/EuroHPC JU, the Ministry of Digital Transformation and FINKI through the VEZILKA project, Grant Agreement No. 101263128. Regional recordings: SYNDI project, Ministry of Culture (2021). Contact: info@vezilka.ai

Partners: VEZILKA, AI Factories, EuroHPC Joint Undertaking, co-funded by the European Union, FINKI, UKIM, Ministry of Digital Transformation