July 2026 Summaries

At a recent San Francisco meetup, experts shared insights on building robust voice agents capable of handling real-world interactions, emphasizing the importance of flexible, cascading architectures that allow customization and optimization of individual components. They highlighted the significance of maintaining a 1–1.5 second latency to ensure natural conversations, as well as the need for layered evaluation processes that combine automated benchmarks with human judgment. Achieving a human-like interaction involves careful turn-taking and context management, with the latter being crucial to maintaining continuity across conversations. The discussion also addressed the challenges of deploying voice agents in production environments, where adaptability and learning from failures become valuable differentiators. Cost management through precise measurement and model configuration adjustments was advised, alongside excitement for self-improving systems that could autonomously enhance their performance without human intervention. The overarching theme was the necessity for accurate transcription as a foundation for successful voice agent functionality, supported by the capabilities of the Universal-3.5 Pro Realtime model.

AssemblyAI's Universal-3.5 Pro Realtime model has been recognized as the only model within Coval's Human Parity Zone, which signifies it matches or exceeds human performance in both accuracy and speed in speech-to-text tasks. Coval's independent, open-source benchmark evaluates models using a standardized methodology, scoring them on accuracy via word error rate (WER) and latency, the time before a model starts responding. The Universal-3.5 Pro Realtime model achieved a 3.40% WER and the fastest response time, outperforming 28 other models, including those from Google and OpenAI. This model's ability to match human transcription performance is attributed to its innovative Context Carryover feature, which enhances accuracy by intelligently applying conversation context, setting a new standard for voice agents in both transcription and responsiveness. Coval's benchmarks are valued for their transparency and reproducibility, using a mix of easy and complex audio samples to reflect real-world use cases, and are available for public verification, ensuring the integrity of the results.

Universal-3.5 Pro has demonstrated superior performance across three independent speech-to-text benchmarks, confirming its high accuracy and competitive latency. The Coval benchmark, known for its rigorous testing conditions that include real-world audio challenges such as accents and noise, ranked Universal-3.5 Pro at the top with a 3.4% word error rate. Similarly, Daily's Pipecat STT benchmark placed it on the Pareto frontier for its balance of semantic accuracy and latency, with a median time-to-final segment of 282 ms. The Hugging Face Open ASR Leaderboard, which evaluates models on clean, scripted speech, also ranked Universal-3.5 Pro near the top. These consistent high rankings indicate the model's capability to handle both pristine and complex, real-world audio effectively, providing significant external validation of its quality and reliability for developers considering voice AI solutions.

For healthcare AI product teams, the decision to build or buy an AI medical scribe like Nuance DAX or Abridge is crucial, with significant strategic implications. Buying off-the-shelf solutions provides immediate value, with ready-to-use features like EHR integration and compliance, but limits control over note formats and specialty support, and locks in ongoing per-seat pricing that can cap margins as the user base grows. Conversely, building an in-house scribe allows for complete customization of the note structure and user experience, potentially offering a strategic edge but requires significant investment in developing robust transcription capabilities, particularly in the speech-to-text layer, which is pivotal to the scribe's reliability and accuracy. Key considerations include how documentation fits into the product's competitive strategy, potential economic impacts of scaling, and the importance of owning critical technology layers to maintain flexibility and control.

This comprehensive guide outlines the fundamental steps involved in integrating a speech-to-text API, focusing on authentication, polling for job status, and parsing JSON responses. It begins by explaining the authentication process, where users send their API key in the authorization header without a Bearer prefix. The guide details the asynchronous nature of transcription jobs, where users submit audio URLs and receive a job ID to poll until the transcription is completed or errors out. Polling involves checking the status every few seconds until it reaches a terminal state. Once completed, the JSON response provides crucial data fields such as the full transcript, confidence scores, and word timestamps, which are essential for features like captions and search functionality. The document also highlights the importance of correctly handling errors, distinguishing between HTTP-level and job-level failures, and provides insights on enhancing API functionality with additional options like speaker labels and language detection. Lastly, it encourages using webhooks to streamline the process by receiving results directly when a job is completed, thus avoiding continuous polling.

The guide discusses the transition from polling to using webhooks for receiving notifications when transcriptions are completed using AssemblyAI. It highlights the inefficiency of polling, which involves repeatedly sending GET requests to check the status of transcription jobs, and contrasts this with webhooks that provide instant notifications without unnecessary requests. The guide provides detailed instructions on setting up webhooks, including configuring the webhook_url, verifying requests, handling retries, and securing the endpoint with custom authentication headers and IP allow-listing. It also emphasizes the importance of responding quickly to webhook notifications and fetching the actual transcript data with a subsequent GET request. Additionally, the guide explains how to attach metadata to webhooks via query parameters and offers advice on testing webhooks locally using tunneling tools. The overall message encourages the adoption of webhooks for production-grade applications that require efficient and reliable transcription processing.

Real-time speech-to-text technology is rapidly becoming the standard for high-value, interactive applications such as voice agents, live captions, and ambient scribes, as it allows for immediate transcription while a conversation is ongoing. Recent advancements have significantly reduced the latency and increased the accuracy of real-time models, like the Universal-3.5 Pro Realtime, making them competitive with batch transcription models. Unlike batch processing, which is ideal for pre-recorded audio, real-time systems must handle complexities such as turn detection, context retention, and live diarization, as they operate without the benefit of processing the entire audio file at once. The shift towards real-time transcription transforms transcripts into actionable data streams for immediate decision-making in systems, highlighting the importance of considering latency alongside accuracy in evaluating speech-to-text models. While batch transcription remains relevant for archived and pre-recorded content, real-time transcription is becoming essential for applications requiring interactive and instantaneous responses, redefining the benchmark for successful speech-to-text solutions.

A tutorial details the secure method for integrating AssemblyAI's transcription service into mobile applications without a dedicated mobile SDK, emphasizing the importance of not embedding API keys within client-side code due to security concerns. It suggests a design involving lightweight clients for iOS, Android, and React Native that record audio and send it to a server-side backend, which securely stores the API key and communicates with AssemblyAI's REST API. This approach not only protects the API key but also provides advantages like request logging, rate limiting, and the flexibility to update models without redistributing the app. The tutorial provides implementation guidelines for the backend using Node or Python and instructions for developing mobile clients in Swift, Kotlin, and React Native, ensuring the API key never leaves the server. Additionally, it discusses setting up a pipeline for file transcription and offers insights into implementing real-time streaming transcription using temporary tokens.

Startups building their first voice product have three main paths to consider: a DIY multi-vendor approach, using a platform like Retell or Vapi, or opting for a single API that handles the entire pipeline of speech-to-text (STT), language learning models (LLM), and text-to-speech (TTS). The DIY approach offers maximum control but involves managing multiple vendors and complex integrations, while platforms provide a quick launch with limited customization and potential integration challenges. A single API approach, exemplified by AssemblyAI's Voice Agent API, offers a balance with streamlined integration, predictable pricing, and flexibility without the overhead of managing multiple vendors. Key evaluation criteria include accuracy of speech transcription, latency, developer experience, pricing, and the potential for vendor lock-in. The guide emphasizes the importance of focusing on input accuracy as errors can lead to misinterpretations, and it highlights the benefits of a single API for code-forward startups seeking efficiency without sacrificing control.

The guide discusses building a real-time voice agent using Pipecat, an open-source Voice AI framework, in conjunction with AssemblyAI's Universal-3.5 Pro Realtime model as the speech-to-text engine. Pipecat's modular design allows for the easy swapping of components, and AssemblyAI's model, known for its accuracy, offers features like punctuation-based turn detection, Context Carryover, and keyterm prompting, which enhance live conversation capabilities. The integration is streamlined by AssemblyAI's first-party Pipecat plugin, eliminating the need for manual WebSocket configurations. The Universal-3.5 Pro Realtime model supports 18 languages and offers server-side noise suppression, making it suitable for various applications, including medical and legal contexts. It also provides options for speaker labeling and fine-tuning turn detection settings. The tutorial provides a step-by-step approach to setting up the voice agent, including prerequisites and deployment instructions, with the possibility of testing on Pipecat Cloud, emphasizing the cost-effectiveness of AssemblyAI's service at $0.45 per hour with no minimums.

The text outlines a comprehensive guide on building an AI phone agent using Twilio Voice and AssemblyAI Universal-3.5 Pro Realtime for real-time speech-to-text processing. It describes the integration process where Twilio streams 8 kHz μ-law audio directly into the AssemblyAI model, eliminating the need for resampling. The architecture involves using a WebSocket to handle incoming calls and convert audio into text via the AssemblyAI model. The guide emphasizes the use of punctuation-based turn detection and offers tips for improving transcription accuracy, such as handling noisy environments with Voice Focus and managing turn silences to avoid premature call terminations. It also discusses practical setup instructions, including acquiring necessary API keys, configuring Twilio, and deploying using platforms like Railway or Render. Additionally, the text highlights the cost considerations and advantages of using AssemblyAI's model for phone agents, such as its compatibility with Twilio's audio format and its effectiveness in accurately transcribing conversations in real-time.

Vapi is a managed voice platform that simplifies the creation of voice agents by handling telephony, turn-taking, and orchestration, while supporting over 14 speech-to-text providers. The guide focuses on using AssemblyAI's Universal-3.5 Pro Realtime model as the speech-to-text engine within Vapi, emphasizing its market-leading accuracy for voice agents, particularly in recognizing complex alphanumeric entities and domain-specific vocabulary through keyterm prompting. With a low word error rate of 6.99% on Pipecat's benchmark and a latency of around 150 ms, this model is ideal for real-time applications. Setting up a Vapi agent using AssemblyAI involves minimal configuration: adding an API key, selecting the transcriber and model, and optionally customizing for multilingual support. The Universal-3.5 Pro Realtime model supports 18 languages and includes features like Context Carryover, enhancing its suitability for varied conversational contexts. The model is priced at $0.45 per hour for transcription, with no minimum commitment, and Vapi's platform and other provider fees are billed separately. The transition from older model identifiers to "universal-3-5-pro" is scheduled by September 2026.

This tutorial provides a comprehensive guide on building a real-time voice agent in Node.js using the AssemblyAI Universal-3.5 Pro Realtime model for speech-to-text, without the need for Python or heavy framework dependencies. The setup includes two modes: a terminal agent that uses mic input and plays TTS audio in the terminal, and a browser server utilizing Node.js WebSocket with a user interface. The AssemblyAI model offers features like punctuation-based turn detection, context carryover, and mid-session keyterm prompting, enhancing the transcription process and eliminating the need for a separate VAD library. The tutorial emphasizes the model's efficiency in handling real-time conversations with low word error rates (WER) and provides detailed instructions on connecting to the AssemblyAI WebSocket, streaming audio, and fine-tuning turn detection. It also highlights the ability to update conversation context and keyterms mid-session without needing to restart the connection, thereby optimizing the real-time voice agent's functionality.

In evaluating systems that provide speaker-labeled transcripts, particularly for applications like meeting notes or sales coaching, the common metric Diarization Error Rate (DER) is argued to be less effective than cpWER for optimizing performance. The text highlights that DER can be misleading due to the complexities involved in its measurement and suggests that cpWER offers a more accurate reflection of system efficacy in identifying who said what in audio recordings. The discussion is supported by real-world audio examples, emphasizing the practical implications of choosing the appropriate metric.

Voice agents, used in tasks like drive-through orders and patient intake, often face issues when deployed due to the unpredictability of real-world environments, such as noise and accents, which aren't captured in staging tests. To address this, a process using existing logs and AssemblyAI tools has been developed to preemptively identify and solve these issues. This involves a multi-step diagnostic pipeline that retranscribes audio, summarizes interactions, and scores them against a diagnostic rubric to catch problems before users do. By employing Render for task automation, this method effectively surfaces configuration mistakes, such as the misconfiguration of speech turn-taking settings, thus preventing user-reported issues and allowing teams to focus on more complex challenges. This proactive approach offers a scalable solution without the need for a full observability platform, leveraging existing data and a few additional tools to create a self-service diagnostic system.

Time to first token (TTFT) is a crucial metric for evaluating the responsiveness of voice agents, measuring the elapsed time from when a user stops speaking to when the system produces the first usable output. Unlike word error rate (WER) or average latency, TTFT captures the immediate experience of a conversation, as it determines the responsiveness that users perceive. The article explains that TTFT is essential because it dictates how quickly a voice agent can respond, with the end-to-end pipeline involving speech-to-text (STT), language model (LLM), and text-to-speech (TTS) stages. Accurate measurement involves recording realistic audio clips, marking the end of user speech and the receipt of the first token, and reporting percentiles to understand typical and tail-end experiences. The piece emphasizes that while WER addresses accuracy, TTFT is pivotal for ensuring a voice agent feels alive and responsive, making it a vital consideration for speech-to-text evaluations.

The text discusses the differences between synchronous (sync) and asynchronous (async) transcription methods, focusing on their use cases, speed, and cost. Async transcription involves submitting an audio file, which is processed in the background, with the transcript being collected later, making it suitable for lengthy files where depth and thorough speech understanding are needed. In contrast, sync transcription allows an immediate return of a transcript within the same HTTP response for short clips, typically under two minutes, making it ideal for scenarios where speed is critical, such as when a person or agent is waiting for the results. Both methods use the same underlying model, ensuring transcription accuracy remains unchanged, but they differ in their delivery approach. Async is optimized for cost and completeness, while sync prioritizes speed, offering a "faster than real-time" solution for short audio clips.

Sophisticated voice-agent teams are increasingly opting for the sync HTTP pattern over WebSockets for speech-to-text (STT) processing, which involves sending audio as plain HTTP requests for each conversational turn. This approach is particularly appealing for teams that already manage their own voice-activity detection (VAD) and aim for a uniform stateless architecture that is easy to scale, load-balance, and debug. Although this pattern introduces slightly higher per-turn latency compared to streaming, it simplifies the architecture by removing the need to maintain a persistent connection and manage streaming state. This design is especially suitable when teams want to own the orchestration and maintain control over the integration of STT, language models (LLM), and text-to-speech (TTS) services. However, for those who prefer a managed pipeline, the Voice Agent API offers a comprehensive solution with built-in turn detection and other features.

The tutorial provides a comprehensive guide to building a dictation app, named Blurt, using AssemblyAI's Sync API, emphasizing the importance of minimizing perceived latency to create a seamless user experience. It explains that the initial lag in dictation apps is often due to the infrastructure rather than the speech model itself, and introduces a technique to reduce this lag by pre-warming the connection. The tutorial details steps to capture audio with Python, send it via HTTP to the Sync API, and quickly receive a transcript, highlighting the benefits of a single HTTP request over more complex streaming solutions for short, user-triggered speech. By focusing on optimizing tasks that occur after the user stops speaking, such as DNS resolution and TCP connection setup, the tutorial demonstrates how to make the app feel more responsive. It concludes with considerations for future enhancements and situations in which different API types, like streaming or async, might be more appropriate depending on the use case.

The discussion centers around the strategic decision of whether to use built-in turn detection from streaming speech-to-text (STT) providers or to implement custom voice-activity detection (VAD) for voice agents, highlighting the benefits and trade-offs of each approach. Teams that have fine-tuned their own VAD systems may prefer to bypass the built-in turn detection of streaming models, opting instead for the speed and control of a synchronous HTTP approach to return transcripts quickly and accurately after a turn ends, minimizing latency in the conversation flow. This enables these teams to keep control over the conversation dynamics and maintain high transcript accuracy, which is crucial for the effectiveness of language models in processing user input. Meanwhile, teams that prioritize ease and speed of deployment might choose built-in solutions, which offer integrated turn detection and mid-utterance streaming capabilities, albeit with less control over the endpointing process. The decision ultimately hinges on whether the priority is maintaining control over the conversation loop or achieving quick deployment with pre-configured solutions.

AssemblyAI has introduced the Sync API, a new service designed to efficiently transcribe short audio clips with minimal latency. Unlike the existing Async and Realtime APIs, the Sync API allows users to send an audio clip via a single HTTP POST request and receive a completed transcript in response within approximately 134 milliseconds. This development addresses the specific needs of short-duration audio tasks such as dictation, voice commands, and voice agents, eliminating the need for job queues or WebSocket connections. The Sync API, powered by the Universal-3.5 Pro model, boasts a low word error rate and provides high accuracy without the need for additional model management or integration overhead. It supports audio up to two minutes in length, offers prompt steering for various languages, and is priced at $0.45 per hour, making it a competitive option for developers requiring quick and precise audio transcription.

The text discusses the challenges and solutions for achieving high transcription accuracy with speech-to-text technologies in difficult audio environments, such as noisy call centers with overlapping speakers and filler words. It highlights that while most speech-to-text demos showcase clean audio, real-world scenarios often involve messy audio where the accuracy is truly tested. The Universal-3.5 Pro model is designed to handle such hard audio, providing significant improvements in entity accuracy, especially for critical tokens like email addresses and medical terms. Instead of pre-cleaning audio, server-side tools like Voice Focus are recommended to isolate primary speakers, and multichannel transcription is advised to prevent cross-channel bleed. Additionally, the use of keyterms prompting can help the model recognize specific domain vocabulary, while contextual prompting allows customization of how filler words and disfluencies are transcribed. The text emphasizes the importance of tuning transcription models against one's own challenging audio samples to ensure the best performance in practical applications.

When it comes to batch transcription of pre-recorded audio, AssemblyAI's Universal-3.5 Pro model stands out for its superior accuracy in capturing critical entities like names, numbers, and domain-specific terms compared to Deepgram. This model not only offers a 37% improvement in missed entity rate for emails and significant gains for medical terms and other high-stakes tokens but also provides the fastest turnaround time among AssemblyAI's offerings, transcribing a 5-minute file in approximately 9 seconds. The pricing is straightforward at $0.21 per hour of audio, with no minimums and volume discounts, and the model supports 18 languages with the possibility of automatic fallback to Universal-2 for 99-language coverage. Migration from Deepgram to AssemblyAI is simplified to a one-parameter change, making it an appealing choice for production batch workloads where accuracy is crucial, while Deepgram remains a viable option for general conversational audio.

Determining the best developer experience (DX) for voice agent APIs is crucial as it represents a hidden cost in building voice products, affecting everything from implementation speed to long-term maintenance. A comprehensive checklist for evaluating DX includes considering factors like the time to get a first working agent, the complexity of the API surface, documentation quality, the ability to change configurations mid-call, debuggability, billing clarity, and integration with existing frameworks. AssemblyAI's Voice Agent API is highlighted as a strong candidate due to its simplified integration process through a standard WebSocket and plain JSON, allowing for quick setup, mid-stream reconfiguration, and straightforward billing at $4.50 per hour. It also supports popular frameworks like LiveKit and Pipecat, offers native Claude Code integration, and provides a visible transcript for efficient debugging. Ultimately, the best API is one that meets these criteria and can demonstrate its value through a practical, one-afternoon bake-off test that evaluates real-world use cases.

Building a voice agent that can seamlessly transfer to a human involves creating a system where the agent recognizes when it cannot handle a request, such as a refund outside policy or a billing dispute, and initiates a "warm handoff" to a human agent without losing context. This process is achieved using AssemblyAI's Voice Agent API, which utilizes a tool called "transfer_to_human" that triggers when certain escalation conditions are met, as described in the agent's system prompt. The handoff process includes generating a concise summary of the conversation, ensuring the human agent is informed of the situation before speaking to the caller, which relies on an accurate transcription provided by the API's Universal-3.5 Pro Realtime model. During the transfer, the agent uses a "hold" execution mode to remain silent, avoiding awkward small talk, and updates the caller on the status to prevent confusion. The actual phone routing is handled by a telephony provider like Twilio, with AssemblyAI providing guidance for integrating the API with such services, ensuring a smooth transition and maintaining customer trust.

Voice agent architectures can be divided into three main types, each with distinct trade-offs concerning latency, control, cost, and accuracy: the cascading STT→LLM→TTS pipeline, the single speech-to-speech model, and the unified voice agent API. The cascading pipeline offers the highest control and granularity by allowing the choice of best-in-class models for each stage, but it requires integration and coordination of three separate components, resulting in greater complexity and potential latency issues. The speech-to-speech model simplifies the architecture by processing audio input to output within a single model, reducing integration complexity but sacrificing control and transparency over individual stages. The unified voice agent API, such as AssemblyAI's offering, combines the benefits of both by running the full pipeline behind a single connection, offering a balance of control and simplicity, and is particularly suitable for production-grade accuracy without the need to integrate multiple vendors. A critical factor across all architectures is the accuracy of the speech-to-text (STT) layer, as any errors in transcription can propagate through the system, affecting the overall performance and reliability of the voice agent.

Conversation context in voice AI refers to the practice of providing a speech-to-text model with both sides of a dialog—what the agent just said and what the user has already said—which significantly enhances transcription accuracy for short replies and spelled-out entities such as emails, names, and numbers. This approach helps the model anticipate and correctly interpret ambiguous or similar-sounding words by leveraging the context of previous spoken interactions, thus targeting the areas where voice agents typically struggle the most. The Universal-3.5 Pro Realtime model incorporates this method by maintaining a short memory of the conversation, with the user’s prior speech carried forward automatically and the agent’s most recent reply supplied through parameters. This technique is particularly beneficial when dealing with predictable responses triggered by specific questions, offering a cost-effective solution to improve accuracy without solely relying on model size or benchmark Word Error Rate (WER) metrics.

Universal-3.5 Pro, released by AssemblyAI, is an advanced asynchronous speech-to-text model designed to handle real-world audio challenges, including code-switching across 18 languages, speaker diarization, and contextual prompting for improved transcription accuracy. Unlike traditional systems, it captures code-switched speech natively without configuration, maintaining the integrity of conversations by accurately transcribing each language as spoken. The model excels in complex speaker diarization, providing speaker-annotated transcripts that mirror natural conversation flow, crucial for environments like call centers where interruptions and rapid exchanges are common. Contextual prompting enhances accuracy by allowing users to input domain-specific knowledge, making it particularly effective in specialized fields such as healthcare and call centers. With robust support for a wide range of languages and the ability to integrate real-world context into transcriptions, Universal-3.5 Pro is positioned as a foundational tool for industries that rely on precise and reliable transcription capabilities.

Universal-3.5 Pro Realtime introduces advanced contextual awareness to speech-to-text models, enhancing transcription accuracy by considering conversational context and dynamic prompts. Traditional models transcribe audio in isolation, but this update allows the model to use information about the conversation's participants and content, improving performance in noisy environments where voice agents operate. Key features include contextual prompting, which involves providing detailed natural-language prompts to guide the model's understanding, and conversation context, which enables the model to retain and utilize previous interactions within a session. This contextual approach allows for real-time updates during conversations, such as adjusting prompts based on new information, thereby maintaining accuracy even in challenging audio conditions. The combined use of these features significantly reduces errors, especially in real-world scenarios like customer support, by allowing the model to anticipate and correctly transcribe domain-specific terms and conversational nuances.