Voice AI is benchmarked on read speech. Your users aren’t reading.
23
Every few weeks a new speech model arrives with a headline number: more languages, a lower word error rate, a higher place on a leaderboard. The numbers are real. But look at what most of them are measured on and a pattern shows up: the test audio is, overwhelmingly, people reading.
That matters, because almost nobody talks to a voice product the way they read a sentence aloud.
What the benchmarks actually contain
FLEURS is the reference point for multilingual speech recognition. It covers 102 languages with roughly 12 hours of speech per language, and it is built on the FLoRes-101 machine translation benchmark. Native speakers were recruited to read the translated sentences and asked to record in a quiet environment, on a phone or a computer [1]. That design is what makes FLEURS useful: every language says the same sentences, so results compare cleanly across languages.
It is also what makes it a narrow test. Translated encyclopedia sentences, read in a quiet room, are a long way from a customer explaining a billing problem over a bad line.
The pattern repeats one level up. The multilingual track of the Hugging Face Open ASR Leaderboard, a widely followed public ranking of open speech models, evaluates five European languages on CoVoST-2, FLEURS and Multilingual LibriSpeech. Its own paper classes all three as read speech; the fourth set it lists, VoxPopuli, is parliamentary speeches [2]. Read speech is the easiest to collect, license and transcribe at scale, so it is what the field measures.
What happens off the script
When researchers test the same models on how people actually talk, the scores move.
A 2024 study built a multilingual benchmark from unscripted adult phone conversations. It found a significant drop across state-of-the-art ASR models compared with standard benchmarks such as LibriSpeech and FLEURS, and error rates rose with disfluencies: pauses, restarts, interruptions [3].
Accent is the second gap. In 2020, a study in PNAS tested five commercial speech recognition systems from major technology companies. It found an average word error rate of 0.35 for Black speakers against 0.19 for white speakers, nearly double [4]. More recently, the Afrispeech-Dialog benchmark measured a degradation of more than 10% for African-accented English compared with native accents in long-form conversations, and showed the errors carrying through into LLM summaries built on those transcripts [5].
Put the two together and you get the conditions most of the world’s voice users live in: spontaneous speech, an accent the benchmark doesn’t contain, an ordinary phone, a room that isn’t quiet. Often a second language. Often two languages in the same sentence.
None of this is a criticism of the benchmarks. A benchmark has to be fixed, comparable and clean to do its job. The problem starts when a clean benchmark score is read as a forecast of production quality.
Why this is a data problem, not a modelling problem
Models learn the distribution they are shown. If training and evaluation data are mostly read, mostly quiet and mostly from a handful of accent groups, a model can be excellent on that distribution and still fragile everywhere else. The leaderboard won’t tell you, because the leaderboard is drawn from the same distribution.
Two things follow for teams shipping voice products to real users.
Evaluate on the speech you will serve. Before trusting a headline score, build a test set that looks like your users: their accents, their languages, their devices, their rooms, unscripted. A few hours of this tells you more about launch quality in a new market than a leaderboard position.
Train on it too. Fine-tuning on spontaneous, accented, real-environment speech is how you close the gap you just measured. That data is harder to get than read speech, which is exactly why it is scarce.
What that data looks like
This is the gap Silencio was built for. Our contributors record on their own phones, in their own homes, streets and workplaces, in their own languages and accents. Each recording carries verified speaker metadata (country of birth, first language, self-reported accent, age band, gender and device) and a consent record for every speaker.
Two examples:
English as it is actually spoken. Our public Accents of English sample has 282 speakers born in 35 countries: 513 clips and 9.3 hours, each labelled with the speaker’s country of birth, first language and accent [6]. Behind it sits a catalogue of more than 100,000 hours of accented English off the shelf, across read monologues and free speech.
French beyond France. Our Accents of French sample covers speakers born in 9 countries, most of them speaking French as a second language [7]. The full catalogue holds more than 10,000 hours of accented French.
Across the network, around 2.5 million contributors in roughly 180 countries give us access to about 350 languages, and around 500,000 hours of single-speaker speech are available off the shelf. Where a language, accent or code-switching pair isn’t in the catalogue, we collect it to spec, with native speakers in the places it is spoken.
The question to ask
Next time a model tops a leaderboard, ask one question before you plan a launch around it: who was speaking, and were they reading?
If the answer is “people reading in a quiet room”, you know what you still need to test. Our samples are on Hugging Face [8], and we’re happy to scope an evaluation set with you.
Sources
Conneau et al., “FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech”, 2022. https://arxiv.org/abs/2205.12446 (collection method: https://simran-khanuja.github.io/assets/fleurs_slt.pdf)
Srivastav et al., “ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation”, arXiv 2510.06961 (v4, Mar 2026). https://arxiv.org/abs/2510.06961
Maheshwari et al., “ASR Benchmarking: Need for a More Representative Conversational Dataset”, 2024. https://arxiv.org/abs/2409.12042
Koenecke et al., “Racial disparities in automated speech recognition”, PNAS 117(14), 2020. https://www.pnas.org/doi/10.1073/pnas.1915768117
“Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond”, 2025. https://arxiv.org/abs/2502.03945
https://huggingface.co/datasets/SilencioNetwork/english-accents-speech
https://huggingface.co/datasets/SilencioNetwork/french-accents-speech