When low word error rate is not enough: Measuring entity accuracy in ASR
Blog post from Hume
A Real World VoiceEQ evaluation of 45 speech-recognition systems found that low word error rate (WER) does not necessarily mean critical details are transcribed correctly: systems with nearly identical WER differed by seven percentage points in entity accuracy, and even the top system, ElevenLabs Scribe v2, correctly captured only 81.7% of named entities. The English benchmark covered 3,292 recordings, 3,257 entities across 27 categories, and speakers from 25 countries; it counted an entity as correct only when its written form matched, aside from case, punctuation, and spacing. Results varied substantially by system and entity type: accuracy averaged 44% for person names across the five leading systems, while alphanumeric strings averaged 53% across all systems, and foreign-origin entities reduced accuracy by an average of 14 percentage points. The findings show why entity-specific evaluation can reveal failures that sentence-level WER obscures and help developers identify where targeted data is needed. The accompanying Hume ASR Entity Dataset contains about 800 hours of speech in eight languages, with more than 350,000 tagged entities across 28 categories.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.