February 2026 Summaries

A new benchmark has been released to evaluate Speech-to-Text (STT) providers, focusing on transcription latency and semantic accuracy for real-time voice agents. This benchmark aims to assess STT performance in terms of how quickly and accurately a voice agent can transcribe spoken inputs for language model processing, emphasizing that transcription accuracy should prioritize conveying user intent over perfect word-for-word transcription. The benchmark analyzed various STT services on real-world audio samples, highlighting the trade-offs between speed and accuracy, and introduced the concept of Semantic Word Error Rate (WER) to better measure transcription quality for voice AI applications. The results showed that while latency varies significantly among services, the overall accuracy of STT providers has improved dramatically, with three services—Deepgram, Soniox, and Speechmatics—standing out for balancing speed and accuracy. The importance of P95 latency, which reflects the worst-case latency experience, was emphasized, alongside the consideration of finalization support and turn detection for optimizing voice agent interactions. The benchmark tool is available as an open-source utility for developers to test and improve their STT configurations.

The NVIDIA DGX Spark is a compact AI supercomputer designed for AI inference and training, featuring a unique architecture with 128GB of unified memory shared between its Arm CPU and NVIDIA Blackwell CUDA cores. This architecture allows it to handle larger models than typical consumer GPUs. The article explores the experience of training the open-source Smart Turn model on the DGX Spark, a task that was previously done on x86_64 devices, and highlights the necessity of compiling certain library dependencies for the Spark's Arm architecture. Training performance on the Spark, which involved adjusting batch sizes to leverage its extensive memory, was found to be comparable to that of traditional GPUs like the NVIDIA L4 and RTX 5060 Ti. Although Smart Turn is a small model and not memory-limited, the DGX Spark's unified memory offers significant advantages for training larger models and more demanding configurations. The process was streamlined, with minimal changes needed to existing scripts, and it is expected to become even simpler with future software updates.

A new benchmark has been released to evaluate the performance of large language models (LLMs) and speech-to-speech models in lengthy, multi-turn conversations, specifically for voice agent applications. This benchmark assesses tool calling, instruction following, and factual grounding, addressing the rapid growth of voice agent adoption in complex enterprise settings. Despite advancements, the best-performing models, which now score 100% on this benchmark, are too slow for practical voice agent use due to latency issues. Most production voice agents currently rely on text-mode LLMs like GPT-4.1 and Gemini 2.5 Flash, although newer models such as AWS Nova 2 Pro also show promising results. The benchmark highlights the capabilities gap between text-mode and speech-to-speech models, but open weights models like Ultravox are closing this gap. The benchmark aims to provide reproducible, open-source tools for evaluating voice AI, with a focus on improving latency and ensuring robust performance across various real-world scenarios. Despite challenges in benchmark design and execution, the initiative seeks to foster collaboration and innovation in the voice AI community, encouraging contributions to refine and expand testing methodologies.