May 2025 Summaries

Evaluating realtime voice-to-voice agents necessitates a distinct approach compared to traditional cascading architectures, as the seamless end-to-end audio streaming lacks the visibility and control over intermediate steps such as speech-to-text, LLM, and text-to-speech. This shift, while enabling lower latency and more natural interactions, requires adaptation in evaluation strategies, focusing on post-hoc analysis, offline detection of safety or quality issues, and audio-in/audio-out simulations due to the absence of text-level simulations. Key challenges include ensuring workflow coverage, tool accuracy, instruction fidelity, and repair behavior, as realtime models may struggle with structured tasks and tool invocation without intermediate text layers. To effectively evaluate these systems, one must employ audio-driven simulation, behavioral instrumentation, and continuous regression testing, tracking tool usage accuracy, instruction clarity, and response naturalness. Coval offers a comprehensive evaluation platform designed to simulate realistic interactions, track structured success, and monitor performance in production, addressing the unique requirements of realtime voice-to-voice agent evaluation.

The enterprise voice AI landscape is evolving from initial excitement to practical deployment, as highlighted by Kevin Wu, founder and CEO of Leaping AI, who transitioned from management consulting to building complex AI workflows. Kevin identifies a significant shift from traditional voice bots, which relied on predefined intents, to systems leveraging large language models (LLMs) that enable natural conversations, though this transition presents challenges for existing players. Leaping AI's approach involves a hybrid architecture that combines LLMs for natural language understanding with deterministic systems for specific tasks, emphasizing the importance of continuous optimization and specialized roles, like Voice AI managers, to ensure quality and efficiency. Kevin underscores the economic viability of voice AI primarily for larger call centers and advocates for gradual deployment starting with high-ROI use cases. His insights reveal that success in voice AI requires not only technical sophistication but also realistic planning, organizational commitment, and the ability to deliver tangible business value by managing the complexities of deploying conversational AI at scale.

A WIRED investigation highlights the critical shortcomings in current AI agent testing, revealing that a 10% failure rate, as experienced by software engineer Jay Prakash Thakur, is a significant barrier to the deployment of reliable autonomous AI systems. Examples like AI agents mishandling complex restaurant orders or HR bots incorrectly approving leave requests underscore the unpredictability and potential risks involved, which could lead to financial, safety, and legal challenges. OpenAI's Joseph Fireman notes the legal implications, as pinpointing responsibility in multi-agent systems becomes increasingly complex. The industry's existing superficial responses, such as adding human oversight or relying on insurance, fail to address the root of these reliability issues. Coval advocates for a different approach, emphasizing the importance of comprehensive AI agent testing through simulations, rigorous evaluation frameworks, and real-time monitoring to ensure reliability and trustworthiness in AI systems. By investing in robust AI testing infrastructure, companies can prevent future failures and maintain customer trust as AI agents are poised to handle a significant portion of customer service interactions by 2029.

As voice AI transitions from pilot projects to essential production systems, Coval and Vapi have announced a partnership that integrates their technologies to enhance the development of scalable, reliable voice agents. This collaboration addresses the increasing challenges faced by companies deploying voice AI, such as meeting latency expectations, managing interruptions, and ensuring continuous quality assurance. Vapi offers a voice orchestration platform that enables the deployment of sophisticated, low-latency voice agents, while Coval provides evaluation, simulation, and testing infrastructure to maintain agent reliability. Their integrated solution allows for automated evaluations, monitoring of critical metrics, and early regression detection, supporting engineering teams, enterprises, and startups in maintaining production-grade reliability and embedding best practices from the outset. The partnership launches with a joint panel event and a special offer for new customers.