Why AI captions rewrite themselves (and other secrets of real-time speech translation)

Real-time speech translation is a streaming pipeline: speech recognition (ASR) turns audio into text as you speak, machine translation (MT) converts that text into the target language, and — for voice output — text-to-speech reads it aloud. Modern systems show translated captions within roughly 1–4 seconds of the words being spoken, which is comparable to the 2–6 second lag of professional human simultaneous interpreters. Accuracy on clear audio now approaches human transcription on standard benchmarks — but noise, accents, and specialized vocabulary can degrade it sharply, and most of that loss is recoverable with a good microphone and a custom glossary. Here's how the whole thing works under the hood, with the actual numbers.

The pipeline: how speech becomes translated captions

Most production systems today are cascaded: separate models chained together. As Google Research describes it, speech translation "systems have usually been broken into three separate components: automatic speech recognition to transcribe the source speech as text, machine translation to translate the transcribed text, and text-to-speech synthesis." For captions, the third stage is skipped — the translated text is the product.

The cascade has a well-known weakness that researchers call error propagation: if the recognizer mishears a word, the translator faithfully translates the wrong word. Academic work on speech translation notes that cascaded systems are "usually criticized for the error propagation between automatic speech recognition and machine translation models" (Xu et al., 2025). This is why the recognition stage — and everything that helps it, like audio quality and vocabulary hints — matters double in a translation pipeline.

End-to-end models that translate directly from audio (Google's Translatotron line, and the newest voice-to-voice systems) avoid that compounding, but as of 2026 the cascaded approach still powers most live captioning products because each stage can be trained on far more data.

Why live captions rewrite themselves

If you've watched live captions, you've seen words appear, then change. That's not a glitch — it's the core mechanism of streaming recognition.

Streaming ASR emits two kinds of results:

• Partial (interim) results — the engine's best guess so far, marked as unstable and "subject to change" while audio is still coming in. Google Cloud's Speech-to-Text even attaches a stability score to each partial, estimating how likely it is to be revised.
• Final results — locked-in text, emitted once the engine has enough context (typically at a pause or phrase boundary). Azure's Speech service makes the same distinction between Recognizing (intermediate) and Recognized (final) events.

The reason for revisions is linguistic: later words disambiguate earlier ones. "I scream" and "ice cream" sound identical until more context arrives. A translation pipeline inherits this behavior — and adds its own, because many languages put verbs or negations in places that force the translator to wait or rewrite. Good caption systems tune this trade-off between showing text fast and showing text that won't change.

How fast is "real time"?

Numbers from vendor documentation and engineering guides, as of 2026:

Component figures from industry engineering guides (Fora Soft, Translated); real-world totals compound to 2–6 seconds in unfavorable setups, with under ~3 seconds generally considered acceptable for live events.

Here's the context that surprises people: human simultaneous interpreters have the same lag. The academic measure is ear-voice span — the delay between hearing and speaking — and studies place it between 2 and 6 seconds, with reported means of roughly 2.7 to 4.7 seconds (UCL research review). Lags beyond ~4 seconds are associated with accuracy loss in humans, too. A well-tuned AI caption pipeline sits squarely in the human interpreter's latency range.

How accurate is speech recognition, really?

The standard metric is word error rate (WER): substitutions + deletions + insertions, divided by the number of words actually spoken. WER 10% means one word in ten is wrong in some way.

The honest answer to "how accurate?" is: it depends enormously on conditions.

• Clean, clear speech: OpenAI's Whisper paper reports ~2.5–2.7% WER on the LibriSpeech clean benchmark — roughly human-level on that benchmark — while noting that models still make about twice as many errors as humans across more varied real-world datasets.
• Conversational telephone speech: Microsoft's famous "human parity" result hit 5.1% WER on the Switchboard benchmark in 2017 — with Microsoft's own caveat that noisy environments, distant microphones, and accented speech remained unsolved.
• Distant mics, overlapping speakers: the CHiME-6 challenge (dinner-party recordings via room microphones) is the sobering data point — the winning system still posted ~40% WER, and the baseline was ~51%.
• Accents and dialects: a PNAS study of five major commercial ASR systems found an average WER of 0.35 for Black American speakers versus 0.19 for white speakers — a gap traced to training-data coverage. Non-native accents face the same mechanics.
• Mixed-language speech: code-switching (dropping English terms into Korean, Hindi, etc.) remains a major failure mode, with vendors reporting 30–50% relative WER increases versus single-language speech.

The takeaway isn't "AI is 97% accurate" or "AI is unusable" — it's that the same system can be near-perfect in a quiet room with a good microphone and badly degraded in a reverberant hall with cross-talk.

Translation quality stacks on top

The MT stage has its own quality measures — n-gram overlap scores like BLEU and neural metrics like COMET that correlate better with human judgment. For live captions, though, the practical rule is simpler: translation quality is capped by recognition quality. A perfectly translated misrecognition is still wrong, and a mangled product name gets mangled in every target language at once.

What actually improves accuracy

In rough order of impact:

1. Microphone and distance. The gap between benchmark WER (~3–5%) and dinner-party WER (~40%) is mostly acoustics. A headset or a mic within arm's reach beats any model upgrade.
2. One speaker at a time. Overlapping speech is the hardest open problem in ASR — meeting discipline is an accuracy feature.
3. Custom vocabulary / glossary. Every major engine supports biasing recognition toward expected terms: Google's model adaptation uses phrase sets to boost "proper names or domain-specific words," and Azure's phrase lists show the canonical example: "Contoso bank" being recognized as "can't do so bank" until the term is registered. Feeding the system your product names, attendee names, and industry terms before the meeting is the single highest-leverage software fix.
4. Context. Telling the system what the session is about (a keynote on oncology vs. a sprint retro) helps both recognition and translation choose the right senses of ambiguous words.
5. Speaker habits. Moderate pace and complete sentences help — the same things that help human interpreters, who work in pairs and rotate every 20–30 minutes precisely because this is cognitively brutal (AIIC).

Do live captions actually help people follow?

Yes — and not only when translation is involved. A research review in Policy Insights from the Behavioral and Brain Sciences found that "more than 100 empirical studies document that captioning a video improves comprehension of, attention to, and memory for the video," with particularly strong benefits for people watching in their non-native language. Add the accessibility dimension — the WHO counts 1.5 billion people with some degree of hearing loss, projected to reach 2.5 billion by 2050 — and captions stop being a nice-to-have for international meetings.

What this means in practice

If you're evaluating live translation for meetings or events, this article compresses to four questions: How close is the mic to the speaker? Can you load your vocabulary in advance? Is the latency within the human-interpreter range (2–6 s)? And can every listener get the language they need?

That last set is what we built Newey around: it streams your meeting or event audio straight from the browser into a recognition-plus-translation pipeline, shows captions in a floating overlay within the latency ranges above, supports custom glossaries and session context for the vocabulary problem, and translates into 60 languages — up to 3 shown at once, with a QR option so each audience member reads their own language on their phone. It's free during beta, so you can test these accuracy claims against your own meetings.

For the organizer's-eye view of all options — human interpreters, platform built-ins, and AI captions — see How to run a multilingual meeting.

FAQ

Is AI speech translation as accurate as a human interpreter?

Different failure modes. On clear audio with common vocabulary, modern ASR approaches human transcription accuracy on standard benchmarks; in noisy rooms, with heavy accents or mixed-language speech, humans still win comfortably. Humans also compress and prioritize meaning under pressure, while AI translates everything literally — including the parts a human would wisely drop.

Why do captions appear and then change?

Streaming recognizers emit provisional "partial" results immediately, then replace them with "final" results once enough context arrives (both Google and Azure document this two-phase behavior). Later words genuinely change what earlier sounds meant.

What latency should I expect from live translated captions?

Roughly 1–4 seconds under good conditions. For calibration: professional human simultaneous interpreters run a 2–6 second ear-voice span.

What is WER?

Word error rate — the fraction of words the recognizer gets wrong (substituted, deleted, or inserted) relative to what was actually said. It's the standard accuracy metric for speech recognition.

Does background noise really matter that much?

It's the single biggest factor. The same class of models that score ~3–5% WER on clean benchmarks posted ~40% WER in the CHiME-6 distant-microphone challenge. Fix the microphone before anything else.

How do I stop the system from butchering our product names?

Use a glossary/custom vocabulary feature. Registering domain terms before the session is exactly what speech engines' phrase-biasing APIs exist for, and it's the highest-impact fix after audio quality.

---

Related reading: How to run a multilingual meeting · Zoom translated captions · Google Meet translated captions