September 2026 Summaries

As text-to-speech systems expand from single-voice narration into multi-speaker dialogue, evaluation must measure not only transcription accuracy but also speaker distinction, vocal consistency, natural transitions, conversational timing, and independent emotional expression. Researchers created 48 two-speaker scripts covering similar voices, interruptions, contrasting emotions, and extended conversations, then used blinded human raters to assess five Gemini TTS models on five corresponding dimensions. Gemini models generally performed well in emotional containment, voice stability, and seamlessness, while speaker separation remained the principal weakness, particularly for same-gender pairs; mixed-gender pairs scored much higher, and male-male pairs were often most difficult. Gemini 3.8 Flash showed substantial gains in speaker separation over earlier Flash models, though other models slightly exceeded it in seamlessness, illustrating trade-offs hidden by overall scores. Comparisons with ElevenLabs, Higgs Audio, and VibeVoice indicated that same-gender voice separation is a broader industry challenge, while systems can vary markedly between separating voices and creating convincing conversational dynamics. The evaluation framework is intended to help developers identify specific weaknesses, track progress across model generations, and design more targeted improvements for emerging voice-AI capabilities.

Hume’s Real-World VoiceEQ benchmark evaluates text-to-speech systems on practical qualities beyond single-sentence audio, including long-form stability, expressivity, naturalness, speaker identity, multilingual performance, controllability, and voice replication. Its new Frontier Metric combines expressivity and reliability, reflecting a trade-off in which highly expressive models often sacrifice consistency; under this measure, Google’s Gemini 3.8 Flash leads by pairing top expressivity with strong reliability. Gemini 3.8 Flash and Flash-Lite rank first and second overall among 34 models, with particularly large improvements in extended long-form stability, while Flash also leads in voice design and style-tag adherence. Flash-Lite’s voice replication performance is closer to the field average, and Flash has relative weaknesses in producing younger-sounding voices and controlling volume. Hume argues that public benchmarks provide broad comparisons, while private, held-out evaluations through its Kairos platform help research teams diagnose specific model failures, tailor tests to real use cases, and measure improvement without allowing models to train on benchmark materials.

Hume has introduced a Voice Controllability Leaderboard that evaluates 17 text-to-speech systems on their ability to create voices from descriptions, alter delivery through prompts or inline tags, and match voices to practical roles, using tens of thousands of human judgments alongside standardized acoustic measurements. The evaluation finds that no model leads all aspects of controllability: ElevenLabs v3 performs strongly in voice design, including accents, age, texture, multilingual capability, and role fit, while Google’s Gemini models lead tone, emotion, and instruction-following tests, and Inworld performs well in prosody, several accent categories, vocal bursts, podcasts, and therapist roles. Across models, gender control is highly reliable, but age, certain voice textures, authentic regional and multilingual accents, volume control, sarcasm, fear, disgust, and combinations or late placement of inline tags remain challenging. Results also show that naturalness and suitability for a requested role can diverge, and performance varies substantially by use case, with some lower-ranked models performing competitively in specialized categories. Hume argues that human listeners are essential for these assessments because language-model judging may not consistently reflect human experience or may be affected by evaluation data leakage.

Hume has introduced the Voice Replication Leaderboard, part of its Real World VoiceEQ Benchmark, to assess how effectively 11 text-to-speech models clone a speaker’s identity while maintaining audio quality and natural delivery. Models replicated 25 voices spanning standard speech, emotional recordings, and native and non-native English accents using identical prompts, then received human ratings for same-speaker similarity, quality, and naturalness alongside an objective speaker-embedding similarity score. OpenBMB VoxCPM2 led human-rated speaker similarity, LongCat-AudioDiT-3.5B achieved the highest objective similarity, Cartesia sonic-3.6-beta led naturalness, and Inworld TTS-2 led audio quality, demonstrating that no system was best in every category. Results also showed substantial differences by reference type, with expressive speech posing particular difficulty for some models, while quality ratings were generally high and less variable than identity preservation. Hume argues that developers and product teams should evaluate models across separate dimensions and with deployment-relevant voices, accents, emotions, and conditions, combining human judgment with objective metrics rather than relying on a single overall score.