Voice Standard Bench: how we grade LLMs on voice post-processing tasks

AssemblyAI’s open-source Voice Standard Bench evaluates LLMs for voice post-processing using real call transcripts and synthetic dictation data across 30-second, 120-second, 900-second, and 3,600-second transcript lengths. It measures summarization, information extraction, text correction, and dictation, combining quality scores with cost and latency metrics, including total response time and time to first token, to recommend models available through LLM Gateway. Summarization is judged for faithfulness, coverage, and conciseness; extraction for groundedness and completeness; correction for faithfulness and readability improvements; and dictation through mechanical measures such as exact match, word error rate, punctuation, capitalization, and formatting accuracy. To improve reliability, two independent LLM judges score subjective tasks, researchers manually reviewed judgments and refined prompts, and text-correction references were removed after they were found to contain errors. Current results identify models such as GPT-6 Astra, GPT-6 Luna, Opus variants, Gemini Flash, and gemma-4-31b as leading options for different balances of quality, price, and speed, while emphasizing that live rankings can change. Users can download the benchmark from GitHub to test models on their own transcripts or compare models through AssemblyAI’s API and playground tools.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.