May 2026 Summaries

AssemblyAI's Voice Agent API is designed as a unified pipeline for coding agents, offering an alternative to the traditional multi-vendor setups typically used in the industry. This approach streamlines the development process by integrating speech-to-text, language model reasoning, and text-to-speech within a single system, reducing the complexity and coordination required when using separate vendors. The API emphasizes coding agent interaction over visual interfaces, allowing developers to build, modify, and deploy voice agents more efficiently by focusing on writing and owning code. AssemblyAI's design choices, including a single WebSocket connection and a simplified API surface, aim to improve reliability and ease of use, making it particularly suitable for applications like customer support, appointment scheduling, and sales training, where AI-driven voice interactions can effectively replace human involvement.

Building a voice agent involves navigating complex technical challenges, particularly in managing the coordination of technologies like speech-to-text (STT), language models (LLM), and text-to-speech (TTS). Two primary approaches are discussed: a full DIY stack, which involves selecting and integrating separate components for each function, allowing for deep customization but requiring significant time and expertise, and a streamlined single-WebSocket method using an API like AssemblyAI's Voice Agent API, which integrates these components behind a single endpoint for faster deployment but with less control. The DIY route can take four to eight weeks, offering complete control over each layer, which is advantageous for teams needing specific customizations or compliance requirements. In contrast, the API approach allows for rapid deployment, often the same afternoon, making it ideal for teams focused on vertical-specific applications rather than voice infrastructure itself. Both paths ultimately lead to a functional voice agent, with the choice depending on whether speed or customization is more critical to the team's goals.

In the context of production voice agents, standard benchmarks that measure word error rate (WER) often fail to capture the critical nuances of real-world usage, where entity accuracy—focusing on specific values such as names, account numbers, and medication names—is paramount. Errors in these areas can lead to significant issues, as voice agents misinterpret crucial information that downstream systems rely on, thereby compounding across conversation turns. This discrepancy underscores the importance of evaluating speech-to-text models based on their missed entity rate rather than WER, as the latter does not account for the accuracy required in capturing exact values needed for effective operation. Notably, voice agent builders rank speech-to-text (STT) accuracy as the most important factor, even above latency and cost, because the quality of transcripts directly affects the reliability of downstream processes. AssemblyAI's Universal-3 Pro Streaming model addresses this by offering capabilities such as domain promptability and keyterms boosting, which allow the model to adapt to specific vocabularies and contexts, thereby enhancing entity accuracy and reducing errors that could disrupt service in high-stakes environments like healthcare and finance.

The Voice Agent API is a comprehensive, transparent framework designed by AssemblyAI to streamline the creation of real-time voice agents by integrating six distinct processing stages: noise cancellation, speech-to-text (STT) recognition, turn detection, an LLM Gateway, text-to-speech (TTS) synthesis, and session management. This pipeline offers developers clarity and control by providing observable components and allowing live configuration updates, thus addressing the common pitfalls associated with "magic APIs" that lack transparency. The system supports multilingual interactions, prioritizes entity accuracy in voice recognition, and is equipped with advanced turn and interruption detection to enhance conversational quality. While the API is not yet equipped for LLM provider portability and voice cloning, it is positioned for developers seeking rapid deployment over extensive infrastructure control, priced at $4.50 per agent hour. Additionally, the centralized observability feature allows for detailed inspection of conversation events, making it a valuable tool for teams focused on support or sales applications where conversation quality is critical.

Voice agent builders encounter significant challenges, referred to as "production ceilings," when their products face real-world conditions that test the limits of their initial design and infrastructure choices. These ceilings manifest in three main areas: transcription accuracy, enterprise deployment capabilities, and audio processing in noisy environments. Transcription accuracy often falters with accented speech or domain-specific terms that were not part of initial training data, leading to a high entity miss rate. Enterprise clients frequently require self-hosted deployment options for security and compliance reasons, which many vendors fail to offer. Additionally, the lack of context integration in speech-to-text (STT) models can result in inaccurate transcriptions, as context chaining and keyterm injection can significantly improve accuracy. Companies such as AssemblyAI offer solutions to these issues, including self-hosted deployments and context integration features, enabling voice agents to better handle diverse conditions and requirements.

In the context of a multi-vendor voice agent stack, operational complexities and costs are often underestimated during the evaluation phase. While each component such as speech recognition (STT), language model (LLM), text-to-speech (TTS), and orchestration may function well individually, integrating them from different vendors into a single cohesive product presents significant challenges. This includes managing multiple onboarding processes, billing relationships, observability contexts, and failure surfaces, which contribute to an overwhelming operational load. The architecture requires ongoing coordination to handle tasks like interruption management and state synchronization across systems, which can become burdensome and inefficient. In contrast, a unified voice pipeline, like AssemblyAI's Voice Agent API, consolidates these components into a single system, reducing the coordination burden and simplifying the operational process. This approach is particularly beneficial for teams where voice functionality is a feature rather than the core product, allowing them to focus on the primary aspects of their service without being bogged down by the complexities of managing a multi-vendor stack. For teams whose voice infrastructure is central to their operations, the multi-vendor approach remains viable, provided they are prepared for the associated operational investments.

The text discusses the integration of AssemblyAI's Voice Agent API and Universal-3 Pro Streaming within existing voice technology stacks, specifically for developers already using platforms like LiveKit and Pipecat. It emphasizes that developers do not need to overhaul their existing architecture to benefit from Universal-3 Pro's enhanced accuracy; instead, a simple endpoint change suffices. The text outlines two paths offered by AssemblyAI: the Voice Agent API, a fully managed solution ideal for new builds or quick proofs-of-concept, and the Universal-3 Pro Streaming, which integrates into current frameworks, enhancing speech-to-text accuracy at a lower cost. The importance of speech-to-text (STT) accuracy is highlighted, as it impacts downstream processes such as language model reasoning and text-to-speech outputs. The text suggests running parallel tests to evaluate the new model's performance without committing to a full migration, especially for industries requiring precise entity transcription, such as healthcare and finance. It also mentions that AssemblyAI is compliant with HIPAA for processing protected health information, catering to specific industry requirements.

Universal-3 Pro has undergone significant enhancements, making it the most accurate model in the market for speech-to-text tasks, particularly in code-switching, disfluencies, turnaround time, diarization, and timestamps. These updates include a ~19% relative improvement in code-switching benchmarks and ~5.9% improvement in capturing disfluencies, crucial for verbatim transcription workloads. The model now offers the fastest turnaround time in AssemblyAI’s lineup, with up to 34% improvement in latency and more accurate speaker diarization, handling up to 30 speakers for longer audio files. Timestamp precision has also seen substantial gains, especially for non-English content, enhancing its utility for tasks requiring word-level timing accuracy. These improvements are automatically available for current Universal-3 Pro users, making it a superior choice over its predecessor, Universal-2, across various performance metrics.

AssemblyAI's Voice Agent API is designed to work with coding agents rather than traditional visual interfaces, offering greater flexibility and capability for building voice agents. Unlike visual builders that use point-and-click methods to configure agents, coding agents allow users to describe desired functionalities in plain language, which the coding agent then translates into code. This approach eliminates the constraints imposed by predefined interface fields and allows for more complex, customizable solutions. AssemblyAI provides a detailed setup prompt that guides the coding agent through the process of building a voice agent, enabling users to create a fully functional project without needing extensive coding knowledge. The API integrates with popular coding agents like Claude Code, Cursor, and GitHub Copilot, making it accessible to non-developers who are comfortable with basic command-line operations. By allowing users to own and modify the code directly, this method provides superior adaptability and scalability compared to visual builders, which are limited by their design interfaces.

AssemblyAI has developed a Voice Agent API that diverges from the industry standard of using multiple vendor components by offering a unified pipeline designed for coding agents. This approach is based on the premise that a coding interface, rather than a visual UI, provides a more efficient and flexible way to create voice agents capable of real-time spoken conversation. Unlike traditional setups that require developers to integrate separate services for speech-to-text, language modeling, and text-to-speech, AssemblyAI's solution consolidates these functionalities into a single system, reducing complexity and coordination issues. This unified pipeline simplifies the architecture, offering a streamlined process with a single WebSocket connection, one billing relationship, and fewer event types to manage, which enhances reliability and ease of use. The API is particularly suited for applications such as customer support, appointment scheduling, and sales training, where natural, real-time interaction can replace human involvement. AssemblyAI's strategy emphasizes giving developers ownership over the code and the ability to make modifications easily with the help of coding agents, thus moving away from the constraints of traditional visual interfaces.

The blog post outlines a step-by-step guide to building a functional Voice AI agent without needing to write or understand code, utilizing tools like AssemblyAI's Voice Agent API and Anthropic's Claude Cowork and Claude Code. The author, who lacks a coding background, successfully created an AI voice agent named Pit Lane Pete, capable of discussing F1 regulations in real-time with users. The process is broken down into phases, starting with creating a knowledge base and system prompt using Claude Cowork, followed by building the application and its interface with Claude Code. Key considerations such as conversation flow, sensitivity settings, and security are discussed, with emphasis on using descriptive problem-solving to guide the AI coding agent. The post highlights the accessibility of AI tools, enabling non-developers to deploy voice agents for various applications by specifying the desired persona and knowledge base, while the technical complexities are managed by the API and coding agents.

The text provides a detailed tutorial on building an AI voice agent for tier-1 customer support using AssemblyAI's Voice Agent API and Python. It outlines how to create a voice agent capable of handling common customer inquiries such as order status lookups, account verification, and human escalation with conversation context. The tutorial emphasizes the importance of accurate speech-to-text transcription, particularly for alphanumeric data, and the use of the Universal-3 Pro Streaming model to reduce errors. It describes the architecture of the system, including the use of a single WebSocket for speech understanding, LLM reasoning, voice generation, and tool calling. The guide also covers how to handle common failures in voice agents, such as misheard order IDs and dead air during backend calls, by utilizing natural transition phrases. Additionally, it suggests integrating the voice agent with phone systems like Twilio for real-world applications and highlights the benefits of using AssemblyAI's API, including cost-effectiveness and support for healthcare workflows through a BAA for HIPAA compliance.

In 2026, the use of speech-to-speech voice agent APIs has evolved from experimental technology to a mainstream solution for deploying production voice agents, simplifying processes by integrating streaming speech-to-text, language models, and text-to-speech into a single endpoint. These APIs are evaluated based on accuracy, latency, and pricing, with options like AssemblyAI's Voice Agent API leading in accuracy for phone audio and offering a flat-rate pricing model. The guide explores the differences between native speech-to-speech models and chained APIs, highlighting the importance of speech accuracy on real-world audio for the success of voice agents. Developers are advised to carefully assess APIs using real audio scenarios to determine the best fit for applications such as lead qualification, appointment scheduling, and customer support. The choice between using a single API or a chained STT-LLM-TTS pipeline depends on specific needs, such as language model preferences, TTS voice specificity, and data residency requirements.

The tutorial explains how to build a real-time voice AI agent in Python using AssemblyAI's Voice Agent API, which consolidates speech-to-text (STT), language model (LLM), text-to-speech (TTS), turn detection, and tool calling into a single WebSocket connection for $4.50 per hour. This approach simplifies the process by eliminating the need for integrating multiple providers and managing separate APIs, allowing developers to implement a functional voice agent with under 100 lines of code. The guide offers a comprehensive introduction to setting up the API, capturing audio, handling tool calls, and managing interruptions, as well as tips for optimizing latency and deploying the agent in production. The companion repository includes sample code and tools, making it easy to adapt the agent to specific use cases. The Voice Agent API is particularly praised for its streamlined architecture, ease of operation, and ability to handle interruptions naturally, making it an attractive option for developers seeking to implement efficient and responsive conversational AI systems.

The tutorial outlines the process of building a telehealth triage voice agent using the AssemblyAI Voice Agent API, designed to capture patient symptoms, score severity, and route patients to appropriate care levels without diagnosing or prescribing. The system functions like an experienced nurse, capturing symptoms in the patient's words, assessing severity, and determining the level of care needed through a structured protocol, while ensuring that a human clinician makes the final decision. It emphasizes the importance of HIPAA compliance with encrypted audio, BAA-backed deployment, and PII redaction, and highlights the necessity of using a defined triage protocol and clinical review to ensure safety and accuracy. The architecture relies on tool calls for protocol adherence and incorporates a separate Medical Mode API for post-call documentation and billing-grade accuracy, ensuring that red flags such as cardiac or mental health emergencies prompt immediate escalation to a live registered nurse.

The tutorial outlines the process of building an inbound phone voice agent using Twilio and AssemblyAI, emphasizing the integration of Twilio Media Streams with AssemblyAI's Universal-3 Pro Streaming, GPT-4o, and ElevenLabs TTS, all designed to operate within an 800ms response time. The guide details setting up a WebSocket server to bridge Twilio's 8kHz mulaw audio to AssemblyAI, leveraging a language model for tool calling and generating responses, and then streaming synthesized audio back to Twilio. The architecture aims to minimize latency by avoiding audio resampling and supports concurrent calls using AssemblyAI's model, suitable for phone-based agents needing real-time, natural conversation capabilities. The tutorial also discusses deployment considerations and provides the complete Python code and resources for implementation, with a focus on achieving efficient, natural interactions in phone-based AI voice agents.

Building effective multilingual voice agents requires the integration of four key components: speech-to-text (STT), language models, text-to-speech (TTS), and orchestration software, all functioning within strict temporal constraints to ensure natural conversational flow. These components must adeptly manage multiple languages, accents, and real-time language switching while maintaining a response time under one second. The guide emphasizes the importance of accurate automatic language detection, handling code-switching scenarios, and preserving conversational context during language transitions. It highlights the challenges of achieving high word accuracy across diverse languages and accents, emphasizing the need for at least 90% accuracy to prevent compounded errors through the pipeline. The document also outlines the technical architecture, performance requirements, and practical considerations essential for creating voice agents capable of serving global audiences, with use cases ranging from customer support automation to contact center operations, underscoring the need for integration with existing systems and cultural adaptation.

An AI cold-calling agent built using the AssemblyAI Voice Agent API and Twilio can efficiently dial prospects, qualify leads through natural conversation, and schedule meetings, simulating the role of a sales development representative (SDR) at a fraction of the cost. The system is designed to handle high volumes of structured calls, ensuring compliance with TCPA and DNC regulations through a robust compliance gate before any call is made. The Voice Agent API handles speech-to-text (STT), language model management (LLM), text-to-speech (TTS), and tool calls through a singular WebSocket, allowing for seamless integration and reducing configuration complexity. Essential components include a dialer to manage calls and respect time-of-day rules, a compliance gate to avoid legal violations, and a tool dispatcher for actions like booking meetings or updating CRM data. The API's pricing model and high speech accuracy make it cost-effective, especially compared to human SDRs, while ensuring high conversion rates by maintaining conversational quality and structured output reliability.

The tutorial provides a comprehensive guide to building a Python-based AI voice agent for customer support, focusing on handling order lookups, account verification, and the escalation of calls to human agents. Utilizing the AssemblyAI Voice Agent API, the agent operates over a single WebSocket connection, integrating speech-to-text, LLM reasoning, and voice generation functionalities. The guide addresses common pitfalls in voice agent development, such as speech-to-text errors and awkward pauses during backend tool calls, and offers solutions like using the Universal-3 Pro Streaming model to improve entity accuracy and transition phrases to maintain conversational flow. Developers are encouraged to replace mock data with real CRM or order management systems before deployment, ensuring robust error handling and logging for seamless operation. The tutorial also highlights the Voice Agent API's compatibility with Twilio for phone integration, enabling real-time audio streaming without the need for transcoding, thus facilitating a smooth transition from browser-based prototypes to phone-based customer support solutions.

In a comprehensive comparison of LLM gateways, AssemblyAI's LLM Gateway, OpenRouter, and LLM Gateway.io are evaluated based on pricing, reliability, security, and developer experience, providing insights into which option best suits different AI workloads. AssemblyAI's LLM Gateway is designed for voice AI applications, offering integrated features like speech-native context preservation and unified billing with transcription services, making it ideal for teams already using AssemblyAI for transcription or developing voice agents. OpenRouter, with its wide selection of over 300 models, serves general-purpose LLM applications and those who value model selection, though it comes with a small markup on provider rates. LLM Gateway.io offers maximum customization through an open-source, self-hosted setup suitable for teams needing strict deployment control and custom routing logic. Each gateway offers unique strengths, with AssemblyAI standing out for audio-specific workloads and compliance, OpenRouter for model diversity, and LLM Gateway.io for customization and control.

The tutorial provides a comprehensive guide to building a voice-powered shopping assistant using Python and AssemblyAI's Voice Agent API, which allows for seamless integration of speech-to-text, text-to-speech, and language model responses through a single WebSocket connection. The assistant is designed to manage four key e-commerce workflows: product search, cart management, order tracking, and checkout, each requiring high accuracy in understanding entity specifics such as sizes, SKUs, prices, and quantities. The system emphasizes the importance of explicit confirmation to prevent accidental orders and explores the challenges and opportunities of voice e-commerce, including handling accents, code-switching, and maintaining context in exploratory shopping conversations. The architecture supports various deployment channels, from mobile apps to in-store kiosks and smart speakers, while maintaining a consistent system prompt and tool registry to ensure a personalized and coherent customer experience.

The tutorial provides a detailed guide on incorporating automatic Large Language Model (LLM) fallbacks into a Python-based voice pipeline to ensure resilience against provider outages. It highlights the significance of implementing fallback mechanisms in voice applications, where latency and service disruptions are more impactful compared to text applications. By using AssemblyAI's LLM Gateway, developers can easily set up a fallback chain that automatically switches between different models, such as Claude, Gemini, and GPT, in the event of a primary model failure due to overloads, rate limits, or deprecations, thus maintaining seamless voice sessions. The tutorial emphasizes that fallbacks are crucial for mitigating three main failure modes: provider rate limits, regional outages, and model deprecations, and demonstrates setting up the system with only a few lines of code. The approach ensures that voice agents remain operational and responsive, avoiding dead air during calls, with the added benefit of simplified model management and billing only for the successfully used model.

The tutorial provides a comprehensive guide on building an AI scribe tailored for therapy sessions, leveraging Python and AssemblyAI's advanced tools to ensure accurate transcription and note generation without the need for manual typing. It focuses on capturing therapy-specific audio accurately using AssemblyAI's Universal-3 Pro Streaming and Medical Mode, which are optimized for clinical terminology and speaker differentiation. Post-session, the AI scribe employs the Voice Agent API to produce structured notes in DAP, SOAP, or BIRP formats, allowing clinicians to interact with the scribe via voice commands to dictate additions, ask questions, and finalize notes for electronic health record (EHR) integration. Emphasizing HIPAA compliance, the tutorial outlines necessary security measures such as obtaining a Business Associate Agreement (BAA) and ensuring encrypted storage of transcripts, while highlighting the benefits of freeing clinicians from typing during sessions, thereby improving patient engagement and reducing post-session documentation time.

A Python tutorial demonstrates how to create a responsive voice pipeline using streaming technology, tool calling, and structured outputs through AssemblyAI's LLM Gateway and Universal-3 Pro Streaming. By streaming large language model (LLM) responses sentence by sentence into a text-to-speech (TTS) engine, the latency in voice interactions is significantly reduced, offering a conversational experience with response times under a second. The tutorial details the construction of a Python voice pipeline that handles real-time transcription, streams LLM responses, and incorporates tool calling for executing real-world actions. Additionally, the setup includes using structured outputs for predictable routing and decision-making. This approach ensures that voice agents maintain a natural flow by immediately responding while the LLM continues to process the subsequent sentences, enhancing user experience through reduced perceived response times. The tutorial also explores using structured JSON schemas for machine-readable outputs necessary for downstream processes and emphasizes streaming's role in achieving a seamless voice interaction experience.

LLM Gateway has been relaunched as a streamlined solution for developers to efficiently interact with multiple large language models (LLMs) through a single OpenAI-compatible endpoint. This update introduces features such as automatic fallbacks, real-time streaming with tool calling, structured outputs, and prompt caching, alongside new models from Qwen and Moonshot. It aims to simplify the process for developers who otherwise manage multiple provider accounts and face issues like added fees and potential outages. The gateway supports a curated catalog of models and offers automatic fallback routing to ensure reliability without added latency. Its integration with AssemblyAI's infrastructure enables seamless operation within real-time voice agent pipelines, offering low latency and no extra network hops. The service emphasizes cost-effectiveness by passing through provider costs without additional markup, contrasting with competitors like OpenRouter.

The tutorial outlines how to create a voice agent capable of making outbound calls using the AssemblyAI Voice Agent API and Twilio's Calls API, integrated through a Node.js server. It details the setup of a WebSocket connection to handle speech recognition, language processing, and text-to-speech within a single framework, eliminating the need for separate services. The guide walks through configuring the system to ensure seamless audio transmission between the APIs, with important considerations for legal compliance regarding automated calls. The tutorial also highlights the potential applications of outbound voice agents in various scenarios, such as appointment reminders and customer winback, emphasizing their ability to reach customers effectively and record interactions in real time.

The AssemblyAI Voice Agent API simplifies the creation of real-time voice agents in Node.js by integrating speech recognition, language processing, and text-to-speech into a single server-side solution, eliminating the need for multiple providers. By utilizing a single WebSocket connection, developers can stream audio input from a microphone and receive the agent's audio response without the traditional latency and complexity of multi-vendor pipelines. The API includes features such as neural turn detection, barge-in handling, and tool calling, alongside customizable options like voice selection and turn detection tuning. Developers can quickly set up the system with minimal code, requiring only a Node.js environment, a microphone, and an AssemblyAI API key. The API supports a variety of voices, including multilingual options, and allows for adjustments to better suit specific use cases or environments, such as raising sensitivity settings for noisy areas or including domain-specific vocabulary for improved speech recognition accuracy.

A comprehensive guide on building a multi-user, browser-ready voice agent using Python, LiveKit, and AssemblyAI's Voice Agent API, this tutorial outlines the integration of LiveKit for handling WebRTC transport and the AssemblyAI API for managing the AI pipeline including speech-to-text, language model, and text-to-speech over a single WebSocket. The worker acts as an intermediary between these systems, enabling the creation of a voice agent without needing to develop a separate orchestration layer for STT, LLM, and TTS or constructing a WebRTC stack. The tutorial highlights configuring LiveKit and AssemblyAI to work together, using a single WebSocket for server-side operations, and demonstrates the setup process involving cloning a repository, setting environment variables, and running the worker. It also addresses handling barge-in, tuning turn detection, and scaling for multiple participants, providing troubleshooting tips for common issues like audio quality and tool calling within a LiveKit room, emphasizing the benefits of using these technologies together for efficient voice agent deployment.

AssemblyAI's Voice Agent API simplifies building conversational voice agents by providing an all-in-one solution that handles speech-to-text, language processing, and text-to-speech functions through a single WebSocket connection. This approach eliminates the need to integrate multiple services, reducing latency and potential failure points often associated with traditional multi-service pipelines. The API supports features like neural turn detection, barge-in, and tool calling, while offering a range of customizable settings including speech recognition sensitivity and voice selection. With just a few lines of Python code, developers can set up a fully functioning voice agent, requiring only an AssemblyAI API key and basic hardware like a microphone and headphones. The API's real-time capabilities allow users to receive and respond to audio input seamlessly, making it ideal for various applications, from customer service to healthcare.

This text provides a comprehensive guide on building a server-side voice agent using AssemblyAI's Voice Agent API integrated with Agora's RTC platform, emphasizing the streamlined process of deploying AI voice agents without requiring separate STT, LLM, or TTS services. The integration leverages Agora's low-latency WebRTC capabilities and the Voice Agent API's comprehensive management of speech recognition, reasoning, and text-to-speech, all via a single WebSocket connection. The tutorial details the technical setup, including configuring API keys, running the bot, and handling audio resampling between the differing PCM rates of Agora and AssemblyAI. It highlights the advantages of using the Voice Agent API for neural turn detection, barge-in functionality, and tool calling, and addresses common troubleshooting scenarios related to the Agora SDK and API connectivity issues. The guide also notes the known limitations of the agora-python-server-sdk, such as its beta status and lack of Windows support, recommending the use of Linux or macOS environments for running the bot.

AssemblyAI's Voice Agent API simplifies the creation of AI-driven voice agents by unifying the speech-to-text, LLM reasoning, and text-to-speech processes into a single WebSocket interface. This API allows developers to bypass the complexity of integrating multiple services and instead focus on building voice agents with coding assistants like Claude Code, ChatGPT, and Cursor. The guide provides specific prompts for creating various types of voice agents, such as browser apps and customer support systems, and offers advice on setting up the API environment to ensure coding assistants stay updated with the latest API changes. The API also includes features like neural turn detection, barge-in handling, and tool calling, which are traditionally challenging to implement. Developers are encouraged to use the API's capabilities to efficiently build and customize voice agents, with troubleshooting tips available to address common integration issues. The API is particularly suitable for "vibe coding" due to its streamlined interface and comprehensive server-side handling, making it accessible even to those with limited WebSocket knowledge.

This tutorial provides an in-depth exploration of the Raw WebSocket Voice Agent using AssemblyAI's Voice Agent API, designed for developers who require complete control over voice agent events without an intermediary SDK. Highlighting the protocol's capabilities, the guide describes handling all server events, processing partial and final transcripts, managing tool calls, and maintaining session continuity even after brief disconnections. It emphasizes the protocol's structure, including client-to-server and server-to-client event exchanges, and offers practical advice on starting an agent, handling errors, and troubleshooting common issues such as session timeouts or audio quality. Additionally, the tutorial explains the tool-calling pattern, the importance of accumulating results, and how to resume sessions seamlessly, making it a comprehensive resource for embedding voice agents into custom applications.

This tutorial demonstrates how to create a server-side voice agent using Daily.co and AssemblyAI's Voice Agent API, enabling a bot to join a WebRTC room, listen to participants, and respond with a real voice through a single WebSocket connection. Leveraging the daily-python SDK, this setup simplifies the typical voice-agent stack by integrating Daily.co's WebRTC infrastructure for managing rooms and participants with AssemblyAI's comprehensive AI capabilities for speech recognition, language model processing, and text-to-speech conversion. The process involves configuring a Daily.co room and AssemblyAI API keys, setting up a virtual microphone for publishing audio, and ensuring proper audio resampling between Daily.co’s 16 kHz and AssemblyAI's 24 kHz formats. The system supports multi-participant interactions, telephony integration, and includes options for tuning voice settings and handling interruptions, with troubleshooting guidance provided for common setup issues.

AssemblyAI's Voice Agent API simplifies the creation of browser-based voice assistants by consolidating the speech-to-text, language model reasoning, and text-to-speech processes into a single WebSocket endpoint. This approach reduces latency and complexity by using a single API key, temporary tokens for secure connections, and built-in features such as barge-in handling and tool calling. Users can build a voice assistant app with less than 400 lines of code, utilizing a browser client and a lightweight Node server which ensures the API key remains secure. The API requires audio in 16-bit signed little-endian PCM format at 24,000 Hz and includes options for customizing voice selections and session prompts. Echo cancellation is recommended to prevent the agent from interrupting itself, and tokens must be refreshed for each new WebSocket connection to maintain security. AssemblyAI offers a free tier for development and testing.

AssemblyAI and Deepgram, both offering voice agent APIs at around $4.50 per hour, utilize a cascaded architecture with distinct models for speech-to-text (STT), language models (LLM), and text-to-speech (TTS) processes. AssemblyAI's Universal-3 Pro Streaming model is noted for its higher word accuracy at 94.07% and a lower missed entity rate of 16.7%, compared to Deepgram's Nova-3 model, which has a 92.10% word accuracy and a 25.5% missed entity rate. This disparity significantly impacts the ability of voice agents to perform tasks correctly without needing user repetition. AssemblyAI's voice agent API is praised for its straightforward pricing model, offering flat per-minute billing without concurrency metering, simplifying cost prediction, whereas Deepgram's concurrency metering can lead to unpredictable costs during peak usage. Additionally, AssemblyAI's API supports dynamic mid-conversation updates, enhancing flexibility for applications requiring real-time changes, while Deepgram's approach is more conventional. AssemblyAI is particularly recommended for production environments that prioritize speech accuracy and those in healthcare, with features like Medical Mode for specialized terminology.

The text provides a detailed comparison between AssemblyAI's Voice Agent API and OpenAI's Realtime API, highlighting key differences in architecture, pricing, speech accuracy, and developer experience. AssemblyAI's API, launched in April 2026, uses a dedicated pipeline with specialized models for each task, offering superior speech accuracy with a 94.07% word accuracy rate and a 16.7% missed entity rate. It is also cost-effective at $4.50/hr compared to OpenAI's $18/hr, with a simpler developer experience allowing for quick deployment. OpenAI's Realtime API, utilizing a single multimodal model with broader language support, is suited for those embedded in OpenAI's ecosystem despite its higher cost and slightly lower speech accuracy. AssemblyAI's API is particularly advantageous for healthcare applications due to its Medical Mode and compliance features, making it a strong choice for production-scale voice agent use cases requiring high accuracy and cost efficiency.

AssemblyAI's Voice Agent API and ElevenLabs Conversational AI offer contrasting approaches to developing voice agents, with AssemblyAI focusing on advanced speech understanding and ElevenLabs expanding its text-to-speech (TTS) capabilities into voice agents. AssemblyAI's API, built specifically for production voice agents, boasts superior speech understanding with a 94.07% word accuracy and lower missed entity rates, making it more suitable for tasks requiring precise input capture, such as customer support and clinical workflows. It offers unlimited concurrency, flat-rate pricing, and full API control, allowing for scalable and customizable solutions. In contrast, ElevenLabs provides a managed platform with a focus on TTS quality, supporting over 29 languages but with a cap of 30 concurrent agents, which may limit its scalability and control in production environments. While ElevenLabs offers impressive voice synthesis, its limitations in speech understanding and scalability make AssemblyAI the preferred choice for production-scale voice agents that prioritize accuracy and flexibility.

AssemblyAI's April 2026 recap highlights significant advancements in real-time transcription and voice agent technology, particularly with the introduction of the Universal-3 Pro Streaming model, which boasts an 8.14% word error rate, the lowest among its competitors. This model excels in recognizing structured data like emails and medical terms, making it ideal for voice agents and live audio workflows. The company also launched Medical Mode, an add-on that enhances transcription accuracy for healthcare-related terms, and the Voice Agent API, simplifying the process of building voice agents by integrating speech recognition, language model reasoning, and text-to-speech into a single interface. Additionally, AssemblyAI introduced a range of developer tools, including refreshed documentation, the MCP Server, and the Claude Code Skill, to streamline AI-native workflows. New models were added to the LLM Gateway, allowing users to apply 20+ language models to transcripts with ease, while updates to PII Redaction capabilities improve the accuracy of identifying and removing sensitive information from both text and audio.

A major upgrade to streaming diarization technology has been released, significantly improving speaker attribution in real-time applications, which is crucial for maintaining the accuracy of AI systems and preventing errors like misattributed quotes or incorrect coaching prompts. The updated model, Universal-3 Pro, outperforms competitors such as Deepgram Nova-3 by reducing false-alarm speakers by 42% and phantom turns by 91%, enhancing the accuracy of applications like AI notetakers and live captioning services. With word-level speaker labels, this upgrade allows for precise detection of speaker changes during conversations, thereby improving the overall quality of outputs for downstream AI systems. These advancements address common user frustrations, such as the need to repeat themselves or being interrupted mid-sentence, by providing clearer inputs for language models and more accurate meeting transcripts. This release demonstrates the critical role of accurate diarization in the foundational layers of voice AI systems, ensuring that applications operate more effectively and user interactions feel more like engaging with a competent, attentive human.

The text discusses building a voice research agent using Render Workflows and AssemblyAI to create efficient and responsive voice interfaces for complex tasks. It addresses the challenges of voice interfaces, such as the need for real-time responsiveness despite lengthy background processes like LLM calls and multi-stage searches, which often lead to awkward pauses and brittle sessions. The proposed solution involves separating the voice channel from the orchestration tasks, allowing the voice agent to remain in a lightweight task while background tasks like classification, planning, and synthesis run as discrete, retry-able workflow tasks with independent timeouts and logs. This architecture improves user experience by enforcing a hard 60-second deadline and supporting shape-aware research with Mastra, which classifies questions to optimize search strategies. The system employs a two-way audio tunnel using WebSockets and provides real-time progress streaming and concurrency handling to maintain performance at scale. This approach is adaptable to any voice application requiring background work and emphasizes the importance of independent workflow tasks, shape-aware planning, and hard deadlines.