New benchmark for real-time speech-to-text models pits accuracy against latency

The Artificial Analysis comparison platform has launched a dedicated leaderboard for streaming speech-to-text (STT) models that, for the first time, measures systematically the tension between final transcription quality and the speed at which it is delivered. Unlike classic offline tests, the new methodology evaluates models under conditions that simulate a live conversation: a continuous audio stream, automatic end-of-speech detection, and a requirement to provide an initial partial transcription followed by a final transcription, each with its own error metric and latency.

The central accuracy metric, AA-WER Streaming, is composed of three corpora with different weights: AA-AgentTalk receives 50 % of the weight and represents agent-to-agent dialogues, a scenario that becomes dominant with the rise of voice agents; VoxPopuli contributes 25 % with multilingual spontaneous speech; and Earnings22 completes the remaining 25 % with recordings of formal investor calls, relatively clean English but dense financial terminology. The decision to allocate half of the score to AgentTalk reflects the assumption that the market is moving toward machine-human dialogue rather than pure transcription.

The board separates First Partial Transcription—the partial output that appears after end-of-speech detection—from Final Transcription, which arrives after additional processing. A separate WER and a separate latency (Time to First Partial and Time to Final) are calculated for each stage. The main chart plots final WER against final latency and draws a Pareto line and an “attractive” quadrant that contains models that are both fast and accurate. The same comparison is repeated for the partial stage, and a third chart directly contrasts the two stages to show how accuracy improves when a model is given more time.

Latencies are reported in seconds measured from the moment of end-of-speech (End of Speech), a definition that neutralizes VAD differences between providers. The weighted metric takes a weighted average over all benchmark samples, so long outliers in certain files pull the score upward. This conservative choice penalizes models that stall occasionally, even if they are swift in most cases.

A third dimension of the board is cost: dollars per 1,000 minutes of audio (≈3.7 shekel per 1,000 minutes at current rate). The price enables developers to locate each model in a three-dimensional space of accuracy-latency-cost and to assess whether paying a premium for a proprietary model is justified when an open-source alternative sits near the Pareto line. The board does not publish specific performance figures for individual models at this stage, only the methodological framework, so concrete comparisons will have to wait for the next update.

To date, choosing a streaming STT model has relied on offline benchmarks such as LibriSpeech or on internal ad-hoc tests. The new framework provides a common language: when a provider claims “low WER”, one can ask “at which stage, partial or final? And on which dataset?”. The distinction between the two transcription stages is especially relevant for applications that display live transcription to users during a conversation, where the partial WER is what the user sees and the latency to it determines the real-time experience.