Arabic transcription benchmark
We scored the engine that writes Qarar’s transcripts and six other speech recognisers on the same Levantine Arabic audio, with the same references and the same scorer. These are the measured results, how we got them, and what they do not show.
Full results: Levantine, 2.06 hours
| System | Dialect WER | Folded WER | Raw WER | Flattened to MSA | Written in dialect | Word timings |
|---|---|---|---|---|---|---|
| Qarar (Kalemio engine) | 4.6% | 10.6% | 18.3% | 62 | 210 | Yes |
| Google Chirp 3 via OpenRouter | 5.0% | 9.7% | 10.9% | 255 | 12 | No |
| Meta Muse Voice 1.0 via OpenRouter | 6.3% | 18.2% | 26.2% | 10 | 604 | No |
| Hamsa General V2 | 7.4% | 20.2% | 42.2% | 17 | 679 | Yes |
| ElevenLabs Scribe v2 | 7.4% | 20.4% | 42.7% | 23 | 694 | Yes |
| Whisper large-v3 (Groq) | 17.0% | 29.2% | 41.6% | 56 | 593 | Yes |
| Whisper large-v3 turbo (Groq) | 25.1% | 36.0% | 45.1% | 110 | 481 | Yes |
How to read the columns
- Word error rate (WER)
- The share of reference words the system got wrong: substituted, dropped or added, totalled over all recordings.
- Raw
- Compares the text as written, so spelling that differs from the reference counts as an error.
- Folded
- First removes diacritics and unifies the common alef, hamza, ya and ta marbuta spellings.
- Dialect
- Also counts the Levantine and the formal form of the same word as a match, and does not score numbers, which can be written as digits or as words.
- Flattened to MSA / written in dialect
- Words the system heard correctly but wrote in a more formal form than the reference, or in a more colloquial one. The first number measures how much a system rewrites dialect.
A second run on human transcripts
In a larger run we scored the same engine and other systems on Lebanese Arabic episodes with human-written reference transcripts. Qarar did not come first on text accuracy in this run.
| System | Strict WER | Spelling-tolerant WER |
|---|---|---|
| OpenAI gpt-transcribe | 19.97% | 14.40% |
| Qarar (Kalemio engine) | 21.09% | 15.22% |
| OpenAI gpt-4o-transcribe | 25.69% | 20.00% |
| Google Chirp 3 (ar-LB) | 27.49% | 18.34% |
| Whisper large-v3 (Groq) | 33.21% | 27.48% |
- OpenAI gpt-transcribe had the best text here, but it returns no word timings and no speakers, so a note could not link back to its moment in the recording.
- ElevenLabs Scribe v2 ran out of quota after 26 of the 83 episodes, so it is left out of this table.
Method
The recordings
Run 1 (13 September): seven single-speaker Lebanese videos from one YouTube channel, 2.06 hours. The references are the uploaders’ Arabic subtitles with corrections we track; they are not a fully audited transcript.
We chose the seven videos with the lowest error in the full set, because their subtitles are closest to what was said, so a gap between two systems is real rather than noise in the references.
Run 2 (9 September): 83 Lebanese episodes, 10.4 hours, with human-written transcripts.
The systems
Every system received the same audio bytes. Chirp 3 and Muse Voice were reached through OpenRouter, which returns no word timings and ignores Chirp’s dialect setting, so they were not tested at their best.
Qarar’s row is the transcription engine as deployed on the day of each run.
Scoring
Errors are totalled over all words rather than averaged per recording, so short recordings do not get extra weight.
We wrote the dialect rules against our own engine’s output, and raw WER rewards matching the reference’s spelling. Neither is neutral, so we show all three.
What these results do not measure
- These are the easiest two hours of our Levantine set: Qarar’s folded WER is 10.6% here and 14.5% on the full set. Compare rows with each other; do not quote one row alone.
- Every recording in run 1 has one speaker, so speaker attribution is not tested.
- One dialect, one channel, similar recording conditions. These numbers say nothing about Gulf, Egyptian or Maghrebi Arabic, phone audio or noisy rooms.
- These are not meeting recordings, and meeting products such as Fireflies.ai and Granola were not tested.
Measured, and being measured
- LevantineMeasured
- GulfBeing measured
- EgyptianBeing measured
- Arabic mixed with EnglishBeing measured
- Speaker attribution in meetingsBeing measured
No numbers appear until a run is complete.