June 2026 Summaries

Analyzing and scoring voice agent calls at scale can be efficiently achieved by transforming each call into structured, scored data using transcription and an LLM (Large Language Model) for summarization and evaluation, as facilitated by AssemblyAI's LLM Gateway. Accurate transcription is crucial, as errors can lead to incorrect scoring, and Universal-3 Pro with speaker diarization helps ensure precise, speaker-attributed transcripts. The LLM Gateway, which integrates with models like OpenAI's GPT, Anthropic's Claude, and Google's Gemini, allows for customizable summaries and scoring against predefined criteria such as task success, resolution without escalation, customer sentiment trajectory, and policy adherence. This hybrid approach, combining deterministic Speech Understanding features for objective signals with LLMs for evaluative tasks, ensures consistent and cost-effective scoring, enabling teams to identify and address failure patterns for continuous improvement of the voice agent's performance.

AssemblyAI's Voice Agent API supports six languages—English, Spanish, French, German, Italian, and Portuguese—and facilitates native code-switching, allowing seamless transitions between languages within a single conversation using the Universal-3 Pro Streaming model. Priced at a flat $4.50 per hour, this API integrates speech-to-text, LLM, and text-to-speech processes through a single WebSocket with approximately one second of latency. A critical feature beyond language support is the inclusion of keyterms prompting, enabling the agent to accurately recognize and transcribe specific product names, customer names, and industry-specific jargon, enhancing the overall user experience. While the Voice Agent API covers these six languages, AssemblyAI's broader platform offers support for over 99 languages through Whisper-Streaming for real-time speech-to-text and Universal-2 for post-call transcription, with plans for future language expansion.

Automatic Speech Recognition (ASR) systems face significant challenges in accurately transcribing heavy accents due to data and model-capacity limitations, not because of speaker clarity. Accents cause problems for ASR models because they often lack sufficient training data for diverse accents, leading these models to default to more common pronunciations. Traditional fixes, such as accent-specific models or pronunciation dictionaries, have proven ineffective as they require prior knowledge of the accent and don't address diverse pronunciation. Scaling ASR models, like the Universal-3 Pro with an LLM-based decoder, improves performance by incorporating more parameters and diverse training data, allowing models to hold multiple pronunciations in mind and use linguistic context to resolve ambiguities. This approach, demonstrated by a lower Word Error Rate (WER) on the CommonVoice dataset, provides more accurate transcription of varied accents without needing to pre-select accent types. Techniques like keyterms and general prompting further enhance accuracy by anchoring the model on predictable vocabulary and providing useful context, ultimately making ASR systems more robust against the challenges posed by naturally occurring accent variation in global audio.

Operating batch transcription at scale involves addressing challenges such as throughput, turnaround, concurrency limits, and cost, which become critical when dealing with large volumes of audio files. While per-file latency matters for live calls, throughput—how many hours of audio can be processed per hour—is more crucial for large backlogs. High concurrency, allowing many files to be processed simultaneously, significantly accelerates the process and is more beneficial than model speed for batch operations. Providers like AssemblyAI, which offer unlimited concurrency, can handle large audio volumes efficiently without throttling. Techniques such as splitting long files into chunks for parallel processing can reduce turnaround time significantly. Cost considerations include per-second billing, with no minimums, making it essential to select the right model based on the specific needs of accuracy and volume, rather than solely on speed.

In the discussion about building voice agents without frameworks like Pipecat or LiveKit, the focus is on utilizing AssemblyAI's Voice Agent API, which consolidates speech-to-text, language model processing, and text-to-speech into a single WebSocket connection. This approach eliminates the need for orchestration frameworks when the pipeline doesn't involve multiple vendors, simplifying the architecture by reducing dependencies and operational complexities. The API allows for seamless integration with telephony services like Twilio, which manage the SIP side and deliver audio over a WebSocket, further simplifying the process by bridging two WebSocket connections. This setup is scalable, offering a flat pricing model and various compliance options, making it suitable for enterprise-level deployment without the intricacies of multi-vendor coordination. While frameworks are beneficial for projects requiring specific features like multi-party communication or granular pipeline control, for straightforward voice agents, this streamlined architecture offers an efficient and manageable alternative.

Medical Mode, a feature for AssemblyAI's Universal-3 Pro and Universal-3.5 Pro Realtime transcription models, aims to reduce missed medical entities by approximately 20% without requiring model swaps, re-integration, or new API keys. By simply adding the configuration parameter domain: "medical-v1", users can enhance the accuracy of clinical vocabulary recognition in transcripts, which is particularly beneficial for handling complex medical terms such as drug names. This adjustment does not affect the underlying model's language coverage or entity accuracy, maintaining the existing workflow while improving the precision of medical entities. Medical Mode is available for both pre-recorded and streaming audio in multiple languages, including English, Spanish, German, and French, and incurs an additional cost of $0.15 per hour. The feature aims to minimize downstream errors and manual corrections in clinical workflows, offering an efficient solution for teams transcribing clinical audio at scale, all while remaining compliant with HIPAA regulations.

Building effective voice agents using AI models such as Claude involves more than just generating code; it requires a strategic approach to context and documentation. The process starts with grounding the AI in current API documentation to prevent outdated code issues, as seen with fast-evolving APIs like AssemblyAI. It's crucial to separate the engineering and business logic prompts, initially focusing on the technical setup before layering in business-specific details. Utilizing real call transcripts instead of hypothetical scenarios enables the AI to understand actual user interactions, which enhances the system prompt, key terms, and tools based on real data. The choice between AI models like Claude and Codex is less critical than ensuring access to up-to-date documentation and real conversational data, as these factors are pivotal in developing a voice agent that is accurate and reliable.

Streaming speech-to-text systems often default to English when processing multilingual audio due to a confidence problem rather than a language deficiency. This drift occurs because streaming models must quickly interpret short audio segments, leading to uncertainty and a fallback to English, which is heavily represented in ASR training data. Factors such as short utterances, code-switching, noise, and accents exacerbate this issue. To address this, it is crucial to select models like Universal-3.5 Pro Realtime that support native code-switching and match the language usage of the target audience. Additionally, providing the model with context, setting language biases when appropriate, and anchoring vocabulary with key terms can improve transcription accuracy. Importantly, forcing a single language on mixed-language audio can backfire, so the strategy should be to steer the model with context rather than restrict it.

Building trust in behavioral health documentation hinges on accuracy in clinically significant areas rather than overall transcription precision. Clinicians rely on AI transcription tools that accurately capture specific medical terms, such as psychiatric medication names, which carry significant clinical weight. The Missed Entity Rate (MER) is a more valuable metric than overall Word Error Rate (WER) for these contexts. Furthermore, speaker diarization is critical in behavioral settings to ensure that each speaker's contributions are correctly attributed, especially in multi-party therapy sessions. Privacy and security are equally vital, with HIPAA compliance and PII redaction being necessary to handle sensitive data responsibly. Trust is ultimately established when clinicians feel confident enough in the system's reliability that they no longer feel the need to verify every detail, transforming a tool from a novelty into a trusted part of their workflow.

Keyterm prompting is a method used in real-time speech-to-text applications to enhance the accuracy of transcribing specific, high-value words such as names, jargon, and product terms that are often missed by generic models. This technique involves providing a list of important words and phrases to the model via the keyterms_prompt parameter, which biases the model towards recognizing them during transcription. It operates through two stages: word-level boosting, which occurs live during audio input, and turn-level boosting, which employs phonetic matching to correct sound-alike errors. This approach, particularly effective with AssemblyAI's Universal-3.5 Pro Realtime model, allows for dynamic updates throughout a conversation, improving transcription accuracy even for accented speech. However, overloading the keyterms list can lead to overcorrections, so it is advised to maintain a focused and concise list of genuinely critical terms. The integration of keyterm prompting with other features like conversation and agent context supports the development of precise and reliable speech recognition systems, especially in environments where specific vocabulary is crucial.

The text discusses the inadequacy of Word Error Rate (WER) as a metric for evaluating medical transcription accuracy, highlighting its failure to differentiate between inconsequential errors, like missing filler words, and critical mistakes, such as incorrect drug names. It introduces the Missed Entity Rate (MER) as a superior alternative, focusing on the accurate transcription of clinically meaningful words—such as drug names, dosages, and diagnoses—essential for patient safety. The text provides a comparison of various transcription models, showcasing that a low WER does not necessarily indicate accurate medical transcription, while emphasizing the importance of measuring MER on one's own audio to ensure accurate results. The narrative argues for prioritizing MER over WER in medical transcription to reduce clinical risk, as well as the need for human review despite lower MER scores, to ensure comprehensive quality assurance.

In the evaluation of real-time speech-to-text (STT) models for voice agents, latency is a critical factor, but focusing solely on achieving the lowest latency can be misleading. Instead, it is important to balance speed with accuracy to ensure that the voice agent does not sacrifice correctness for speed. The optimal latency for voice agents should align with human conversational rhythms, around a 200-millisecond gap between turns, while accounting for the entire response loop, including STT, large language model (LLM) processing, and text-to-speech (TTS) conversion. Universal-3.5 Pro Realtime, for example, offers a competitive word error rate while maintaining effective end-of-turn detection at around 300 milliseconds, ensuring that STT does not become the bottleneck in the response loop. Evaluating models should focus on time to complete turn (TTCT) and consider P95 latency, which accounts for real-world conditions where one in twenty interactions may experience delays, rather than just median (P50) latency. By prioritizing a model that balances speed and accuracy, voice agents can deliver a more reliable and user-friendly experience.

Medical transcription in multilingual settings, particularly in clinical environments, faces challenges beyond simple language support, focusing on accuracy in code-switched sentences where patients and clinicians mix languages mid-sentence. This complexity is addressed by Universal-3 Pro and its advanced version, Universal-3.5 Pro Realtime, which handle intra-utterance code-switching across multiple languages, ensuring that transcripts maintain clinical accuracy without needing separate language pipelines or toggles. The models are benchmarked for languages including English, Spanish, German, and French, with Medical Mode achieving a low Missed Entity Rate of 3.2%, representing a significant improvement over other systems. The approach allows seamless transcription of mixed-language utterances common in real-world clinical settings and supports region-specific terminology through keyterms prompting, making the system versatile for various markets without the need for multiple integrations.

Veterinary transcription presents unique challenges due to the complex audio environment of exam rooms and the specialized vocabulary related to species, breeds, and veterinary drug names, which general transcription models often fail to handle accurately. The Universal-3 Pro model, combined with Medical Mode, offers a solution by leveraging its capabilities in human clinical language to cover shared pharmacological and anatomical terms, while keyterms prompting allows customization for vet-specific vocabulary, enhancing transcription accuracy. Keyterms prompting is crucial for accurately recognizing veterinary-specific terms by allowing users to input a list of important names and phrases, thus ensuring that essential information like drug names and breed-specific details are captured correctly. Additionally, speaker diarization helps distinguish between multiple speakers in a noisy clinic environment, providing clear and structured transcripts that attribute statements to the correct individual, which is vital for maintaining the integrity of clinical records.

Clinical AI pipelines are susceptible to error propagation, particularly when speech-to-text (STT) models misinterpret drug names, leading to incorrect SOAP notes and decisions downstream. This issue arises because the language models (LLMs), which generate clinical documentation, rely entirely on the transcripts provided by the STT models without direct access to the original audio. Consequently, any transcription error—such as confusing "hydrochlorothiazide" with "hydrocortisone"—is treated as accurate by the LLM, resulting in coherent yet incorrect documentation. Traditional metrics like Word Error Rate (WER) are insufficient for assessing clinical transcription accuracy, as they treat all words equally without prioritizing critical entities like drug names and dosages. Instead, the Missed Entity Rate (MER) is a more telling metric, measuring how often vital clinical entities are dropped or altered. AssemblyAI's Universal-3 Pro with Medical Mode aims to enhance entity accuracy within clinical transcripts by reducing MER, thereby mitigating error propagation in AI-driven clinical workflows. The service includes features like medical-specific transcription models, speaker diarization, and PII redaction, all designed to ensure the integrity and confidentiality of clinical data.

Universal-3.5 Pro Realtime is AssemblyAI's latest flagship real-time speech-to-text model that emphasizes improved context retention and language handling to enhance transcription accuracy. It allows voice agents to pass questions with context, reducing word error rates significantly by using a rolling memory to keep track of conversations. This model supports 18 languages with mid-sentence code-switching and provides advanced features like voice focus to isolate primary speakers, making it ideal for noisy environments. Universal-3.5 Pro Realtime outperforms competitors in various metrics, such as word error rate and entity error rate, offering a cost-effective solution with add-ons like diarization and voice isolation. It is designed to integrate seamlessly into existing systems, with automatic upgrades for most users and the flexibility to handle large-scale operations without rate limits.

AssemblyAI and Deepgram, both offering voice agent infrastructure at approximately $4.50/hr, present distinct differences in terms of entity accuracy, developer experience, billing models, and streaming speech-to-text (STT) quality, despite appearing similar on paper. AssemblyAI’s Universal-3 Pro Streaming model, topping the Hugging Face Open ASR Leaderboard, outperforms Deepgram’s Nova-3 model in critical areas, such as word accuracy and missed entity rates, making it particularly advantageous for scenarios requiring precise transcription of structured data like emails and phone numbers. Moreover, AssemblyAI’s API provides a more flexible developer experience with features like mid-session updates and intelligent endpointing, which enhances conversation flow by accurately detecting when a speaker has finished talking. In contrast, Deepgram's traditional silence-based voice activity detection (VAD) may result in less natural interactions. For healthcare applications, AssemblyAI offers enhanced transcription for clinical terminology and compliance with HIPAA, a domain where Deepgram lacks equivalent support. Ultimately, while both platforms are viable for voice agent development, AssemblyAI stands out for its superior speech accuracy and predictable pricing, making it a preferred choice for producing reliable and efficient voice agents.

Enterprise voice agents require robust platforms capable of handling high concurrent call volumes, accurately capturing specific entities like account numbers and prescription IDs, and meeting stringent compliance standards. While many platforms appear effective in demonstrations, the real challenge lies in maintaining performance under production conditions with thousands of simultaneous sessions. Key considerations for selecting a platform include speech accuracy, particularly in capturing entities, sub-second latency under load, effective turn detection, enforceable compliance, and scalability without concurrency restrictions. AssemblyAI's Voice Agent API is highlighted for its ability to integrate STT, LLM, and TTS through a single WebSocket connection, offering a flat-rate pricing model and high entity accuracy. Additionally, the platform offers flexibility with a standalone streaming option for teams looking to integrate their own LLM and TTS solutions while maintaining high transcription accuracy. Other platforms like OpenAI and Deepgram are also discussed, each with unique strengths and trade-offs, particularly regarding cost, accuracy, and scalability.

Speech-to-speech voice agents revolutionize the traditional interactive voice response (IVR) systems by enabling natural conversations where users simply speak and the agent responds. This technology relies on two main architectures: the widely used cascaded model, which sequentially processes speech-to-text (STT), large language model (LLM), and text-to-speech (TTS), and the emerging end-to-end model that handles everything in one step. The cascaded approach is favored in production for its accuracy, flexibility, and observability, allowing each component to be optimized and swapped independently. However, it requires careful orchestration to manage latency, turn detection, and error handling, especially in complex interactions. End-to-end models, while simpler and potentially faster, struggle with accuracy and lack transparency, making them less suitable for industries with high precision requirements like healthcare and finance. AssemblyAI offers tools for both approaches, with its Voice Agent API providing a quick start for those seeking a managed solution, while Universal-3 Pro Streaming allows for custom pipeline development with full control over each component.

AI transcription systems often struggle with accurately rendering technical and medical terms due to their specialized vocabulary, which is phonetically complex and rarely appears in general training data, leading to critical errors like incorrect drug dosages. Standard metrics like Word Error Rate (WER) are insufficient for assessing accuracy in these contexts, as they treat all errors equally, failing to account for the impact of specific terminology mistakes. Instead, metrics such as Missed Entity Rate (MER) provide a clearer understanding of how well models handle essential terms. Tools like AssemblyAI's Universal-3 Pro, which offers features such as Medical Mode and keyterms prompting, enhance transcription accuracy by focusing on domain-specific vocabulary. These features allow for customized prompts and vocabulary injection, improving recognition of critical terms necessary for accurate and reliable transcription in fields like healthcare, legal, and technical domains. The challenge of maintaining accuracy extends to non-English languages and poor-quality audio, where strategic use of transcription features and high-quality input can mitigate errors. As AI transcription technology evolves, the ability to customize and optimize for specific domain needs becomes crucial in bridging the gap between AI and human transcription accuracy.

Context significantly enhances automatic speaker labeling by providing additional information that transforms generic labels into specific names or roles, which can be crucial for downstream analysis. AssemblyAI's approach utilizes three types of context: audio context, metadata context, and structural context. Audio context relies on conversational cues within the recording, such as when someone introduces themselves, to infer speaker identities. Metadata context is shaped by user-provided information, like expected speaker count, names, roles, and audio channel mapping, which helps the AI model accurately label speakers. Structural context involves the physical setup of the recording, such as multichannel audio formats that naturally separate speakers. The integration of these contexts allows for more precise speaker identification, which is particularly beneficial in real-world scenarios like contact centers, healthcare, and media production, where accurate speaker attribution is essential for tasks like sentiment analysis, compliance monitoring, and creating searchable transcripts. AssemblyAI's Speaker Identification feature leverages this context-driven approach, replacing generic labels with real names or roles without needing prior voice enrollment, thus enhancing the utility of transcriptions across various applications.

Conversation analytics, also known as conversation intelligence, is an AI-powered technology that transforms unstructured voice data from business interactions into structured, actionable insights. This technology automatically transcribes audio from phone calls, meetings, and other interactions, tagging each speaker and analyzing sentiment, topics, and entities within the conversation. AssemblyAI's API enables the creation of a comprehensive conversation analytics pipeline, offering features like speaker diarization, sentiment analysis, topic detection, entity extraction, and summarization. These tools provide businesses with valuable insights that can enhance customer satisfaction, improve sales outcomes, and streamline operations. The technology's accuracy is crucial, with AssemblyAI's Universal-3 Pro model boasting a 94.07% word accuracy rate, ensuring reliable data for further analysis. By integrating these analytics with business tools, organizations can automate quality assurance, monitor compliance, enhance coaching, and gain real-time insights to drive strategic decisions.

Speaker embedding technology plays a crucial role in speaker diarization, transforming raw audio into speaker-labeled transcripts by determining "who spoke when" in a recording. This process involves using high-dimensional numerical vectors that capture a speaker's unique vocal characteristics, such as pitch and timbre, to distinguish between different voices. The diarization pipeline consists of four main stages: audio segmentation, speaker embedding generation, speaker count estimation, and clustering. Modern approaches employ neural network-based audio embeddings, known as d-vectors, to enhance accuracy, especially in challenging conditions like short utterances and noisy environments. While traditional pipeline-based systems process audio through sequential stages, end-to-end neural systems map raw audio directly to speaker-labeled segments, offering better handling of overlapping speech but less interpretability. AssemblyAI's improved embedding model has significantly advanced diarization accuracy, reducing error rates in adverse conditions by 30% and supporting real-time streaming transcription. The technology is steadily evolving towards speaker fingerprinting, which could allow tracking individual speakers across different recordings and sessions, opening new possibilities for applications in various domains.

SpeakerRevision is an innovative feature that enhances real-time speech processing by providing asynchronous-grade accuracy in speaker labeling at the end of a live stream, thereby eliminating the need for separate asynchronous processing to achieve a clean final transcript. This new message type revises speaker labels with only about 400 milliseconds of added latency, significantly improving accuracy metrics such as DER and cpWER, and reducing false-alarm speakers by 84%. The implementation of SpeakerRevision allows for a unified streaming pipeline that simultaneously supports the live experience and post-call analyses without requiring redundant infrastructure, benefiting various applications like AI notetakers, contact center analytics, and voice agents by delivering both real-time and final transcripts from the same source. This advancement is particularly advantageous for maintaining seamless integration while ensuring high accuracy and reducing operational overhead.

The tutorial provides a comprehensive guide on building a voice agent for IT helpdesk and technical support using the AssemblyAI Voice Agent API. It emphasizes the utility of voice agents in handling repetitive Tier-1 IT support tasks, such as routing issues, looking up internal knowledge bases, managing IT service management (ITSM) tickets, and escalating complex issues to human technicians. The tutorial outlines the setup process, including defining four essential tools—search_knowledge_base, create_ticket, check_ticket_status, and escalate_to_human—and integrating with telephony services like Twilio. It highlights the importance of accurate speech-to-text conversion, especially for alphanumeric data, and suggests using Universal-3 Pro Streaming for improved transcription performance. Additionally, it discusses the option of a bring-your-own-key (BYOK) approach for teams needing specific language models or voice recognition capabilities, while stressing that transcription accuracy is crucial for effective IT support. The tutorial also covers the operational aspects, such as writing system prompts for routing and ensuring containment and escalation precision, ultimately aiming to improve efficiency and reliability in IT helpdesk operations.

Inaccurate transcription can lead to hidden costs that far exceed the apparent per-hour pricing of speech-to-text services, challenging the assumption that the cheapest API is the most economical option. The text details how correction labor, downstream failures, and customer churn due to poor accuracy can significantly inflate the total cost of ownership. It highlights that while cloud platform providers and API-first providers offer different pricing structures, the true expenses often lie in the accuracy of the transcription, which affects labor costs and operational efficiency. The article discusses the varying impact of transcription errors across pre-recorded, streaming, and voice agent uses, noting that higher accuracy can reduce correction times, improve customer satisfaction, and decrease operational overhead. AssemblyAI's models are presented as solutions that, despite higher up-front costs, can offer substantial savings by decreasing errors and associated correction and operational costs, particularly in high-stakes environments like contact centers and voice agents. The text emphasizes the importance of evaluating transcription accuracy using real audio to understand the potential savings from improved accuracy.

The text discusses the limitations of using Word Error Rate (WER) as the sole measure of transcription quality in speech-to-text systems, pointing out that while WER quantifies the accuracy of word transcription, it fails to account for factors that affect the perceived quality, such as speaker mislabeling, formatting issues, and entity errors. These discrepancies create a gap between the system's technical accuracy and users' perception of accuracy, which is critical as it influences trust and user satisfaction. The article highlights the importance of addressing perceived quality by focusing on elements like speaker diarization, audio tag management, and real-time corrections, which can significantly impact user experience even if they do not affect WER. It suggests that the industry's future lies in enhancing perceived quality through newer metrics such as Semantic WER and Missed Entity Rate, which consider meaning preservation and entity-specific accuracy, thereby aligning more closely with user expectations and experiences.

The Voice Agent API by AssemblyAI offers a comprehensive solution for developing voice agents by integrating the entire voice processing pipeline, including speech-to-text (STT), large language model (LLM) reasoning, text-to-speech (TTS), turn detection, and tool calling, all over a single WebSocket connection. Priced at a flat rate of $4.50 per hour, the API simplifies the development process by eliminating the need for multiple service providers and invoices, thus streamlining setup and operation. Key features include adaptive turn detection, which adjusts to a user's speaking pace and context, semantic interruption handling that distinguishes between true interruptions and back-channel affirmations, and the ability to call external tools during conversations. The API supports six input languages and eleven output languages, allowing for multilingual interactions. Developers can easily integrate and customize the API within their applications without needing a dedicated SDK, using standard JSON-over-WebSocket protocols.