September 2026 Summaries

Keyterm prompting can improve transcription of specialized names and jargon, but it may also reduce overall accuracy because supplied terms bias the model throughout an entire transcript rather than only where it is uncertain. Similar-sounding keyterms can overwrite words that were correctly recognized, especially proper nouns, product names, and spelling variants, while long exhaustive lists increase collision risks and dilute the influence of genuinely relevant terms. The text recommends using small, call-specific lists based on available metadata, testing identical audio with and without prompts, measuring full-transcript word error rate rather than keyterm hit rates, validating results separately by language, and retesting whenever lists change. It notes that prompting benefits appear strongest in English, that model limits and overflow behavior should be checked, and that pre-recorded requests can combine general context with keyterms despite restrictions in some integrations. As an alternative, contextual prompting and conversation-aware models are presented as more targeted approaches because they use the current discussion and prior interaction to guide transcription rather than globally favoring a static vocabulary.

Voice-agent launches require post-release operating practices because conversational failures can remain hidden behind normal uptime metrics while callers experience misunderstandings, delays, repeated prompts, and longer calls. The recommended approach is a staged rollout based on representative conversation volumes rather than short time windows, beginning with shadow testing, progressing through small canaries and controlled ramps, pinning each call to one version, and retaining a live control group and previous version for comparison and rollback. Teams should create concise rollback runbooks in advance with measurable thresholds for containment, escalation rates, repeat utterances, P90 turn latency, conversation duration, and cost per resolved call, alongside clear authority, evidence-capture steps, and human fallback procedures. Severity definitions should distinguish complete outages from broad degradation, cohort-specific problems, and cost-only regressions, with quality metrics segmented by factors such as language, accent, intent, and channel. The guidance also emphasizes defining responsibility boundaries between organizations and vendors, evaluating vendor support, versioning, status communication, and rate limits before incidents occur, and preparing for volume spikes through connection backoff, capacity planning, cost alerts, and advance limit increases where possible.

Education platforms evaluating speech-to-text APIs should prioritize performance on difficult student audio, including accented, multilingual, children’s, far-field, and overlapping classroom speech, rather than relying on clean instructor recordings or generic benchmarks. The evaluation should also account for the economics of lengthy lecture archives, seasonal demand spikes, transcription throughput, live-captioning capacity, and the human review costs that can outweigh per-hour API pricing when accuracy is insufficient. Education-specific compliance requirements include FERPA-related data handling, protections involving minors, institutional DPAs, and data residency, while accessibility procurement may require accurate, well-timed captions, speaker labeling, translation alignment, and vendor accessibility documentation such as a VPAT. The text recommends testing vendors with an organization’s hardest real-world audio, verifying policies and support arrangements in writing, and comparing commercial, self-hosted, or hybrid approaches based on actual cohort needs, staffing, infrastructure constraints, and seasonal workloads.

Ambient AI scribes extend the healthcare documentation model to veterinary and legal work by passively recording professional conversations and converting them into structured records through transcription, speaker attribution, and information extraction. Veterinary use requires handling noisy three-way interactions involving clinicians, owners, staff, and animals while preserving distinctions between owner observations, owner interpretations, and clinical findings; it also benefits from tailored vocabulary support for species, drugs, and breeds. Legal applications divide into deposition tools, which require verbatim, speaker-attributed, word-level timestamped transcripts that supplement rather than replace certified court reporters, and client-meeting tools, which extract matter details such as parties, dates, deadlines, instructions, and action items. Across these settings, the most critical technical layer differs: transcription accuracy in veterinary care, speaker attribution in depositions, and reliable structured extraction in legal client meetings. The discussion emphasizes privacy, data residency, review workflows, and provenance links to source audio for high-stakes information, while arguing that rapid, useful initial output may be more important for adoption among smaller professional practices than extensive configuration or feature breadth.

A tutorial demonstrates how to build a roughly 250-line Go service that uses Twilio Media Streams to fork live phone-call audio, AssemblyAI’s streaming speech-to-text API to transcribe it in real time, and the AssemblyAI LLM Gateway to classify the completed transcript as a scam, suspicious, or legitimate call. It explains configuring a public ngrok endpoint and Twilio webhook, preserving Twilio’s native 8 kHz mu-law audio format, and batching five 20-millisecond audio frames into 100-millisecond chunks because the transcription API rejects shorter messages. The application maintains a separate transcript buffer and WebSocket session for each call, stores only finalized formatted speech turns, flushes remaining audio and waits for transcription termination so the final utterance is not lost, then sends the transcript to a configurable LLM with a prompt designed to identify fraud indicators such as urgency, impersonation, requests for codes or payments, and secrecy. It also includes JSON-output fallback handling for models without structured-response support, troubleshooting guidance for common WebSocket, audio-format, and tunnel failures, and suggestions for extending the prototype with mid-call alerts, speaker labels, persistent audit records, or automated intervention.

AssemblyAI has released Universal-3.6 Pro Realtime, a streaming speech-to-text model priced at $0.45 per hour that supports 32 languages with automatic detection and is designed as a direct replacement for Universal-3.5 Pro. Based on the company’s English voice-agent benchmark, the new model reduced short-response errors from 2.65% to 1.45%, normalized word error rate from 5.80% to 5.13%, language-confusion sessions from 0.55% to 0.14%, and background-speech leakage from 3.53 to 2.52 words per 1,000, while maintaining the same reported endpoint latency of 537 milliseconds. It also changes Voice Focus so enhanced audio is used by the recognition model, offers 14 additional languages, improves end-of-turn precision, and adds in-session speaker-label revisions without affecting transcription latency. AssemblyAI cites Pipecat and Coval benchmarks as placing the model competitively among streaming speech systems, though their metrics and test audio differ. Existing users can migrate by changing the model name while retaining current API settings, and Universal-3.5 Pro remains available.

Universal-3.6 Pro Realtime is a streaming speech-to-text model designed for voice agents and call centers, where brief responses, noisy telephone audio, accents, background voices, and spoken entities such as phone numbers can affect downstream actions. Trained on tens of thousands of hours of real voice-agent conversations, it reportedly reduces short-response errors by 45%, language-confusion sessions by 74%, background-speech leakage by 29%, and normalized word error rates by 12% versus Universal-3.5 Pro on the company’s 12,460-scenario English benchmark, with larger gains in heavy noise. The release improves turn detection through training on incomplete speech and introduces entity-aware endpointing, which holds turns open for numbers, IDs, addresses, and similar information, reducing phone-number entity errors while increasing latency only in the slowest cases. An optional voice-focus feature uses filtered audio to suppress a nearby competing speaker, producing substantial gains in tests containing second talkers but modest accuracy costs when no competing speaker is present. The model also adds 14 languages, including Korean and Russian, expands multilingual training, and reports reduced code-switching errors; in the company’s competitive tests it ranks first among 21 systems for English voice-agent word and entity accuracy and code-switched language pairs, while placing third on average across broader multilingual read-speech benchmarks.

MacWhisper, Superwhisper, and Blurt address different speech-to-text needs rather than serving as direct substitutes: MacWhisper is a macOS-focused transcription workstation for files, meetings, subtitles, and batch processing; Superwhisper is a cross-platform dictation app offering local and cloud models plus output-formatting modes; and Blurt is an MIT-licensed macOS dictation tool that uses a hosted API to produce cleaned-up final text. MacWhisper and Superwhisper can operate locally, keeping audio on-device and enabling offline use, while also offering optional cloud features, whereas Blurt always sends audio to the cloud in exchange for fast transcription, multilingual code-switching support, and an integrated language-model cleanup pass. The comparison emphasizes that local processing is preferable for privacy-sensitive, offline, or high-volume transcription work, while hosted processing can improve performance on difficult audio and reduce the effort required to turn spoken language into polished writing. Pricing also reflects these roles, with MacWhisper offering a €64 one-time Pro license, Blurt charging about $0.62 per hour through its API, and Superwhisper using free and paid subscription or lifetime plans with pricing that should be verified directly.

Voice agents that perform well in ordinary conversation but truncate confirmation numbers, account IDs, and other recited sequences may suffer from endpointing errors rather than poor speech recognition, because long pauses between digits can cause turn detection to close before a caller finishes speaking. The proposed diagnosis is to compare transcription failures in conversational speech with those in recited strings: missing sequence tails, especially when general conversation is accurate, indicate premature endpointing. Recommended mitigations include supplying the speech system with the agent’s latest prompt as contextual information, temporarily extending silence thresholds while collecting expected entities, restoring normal conversational settings afterward, and force-ending a turn once an expected value is complete. The piece also recommends updating keyterm prompts by conversation stage for recognizable terms such as names, locations, or product names, monitoring voice activity and speaker-label settings that can affect endpointing, and measuring entity error rates by category rather than relying primarily on pooled word error rate. It argues that evaluations should use complete, real-world conversations with production turn-detection settings, since isolated audio clips assess acoustic transcription but may not expose failures caused by conversational timing and configuration.

macOS includes free, mostly on-device dictation that can be enabled in Keyboard settings and used through a microphone key, shortcut, or menu command in most text fields, with users speaking punctuation and formatting commands aloud. Its principal strengths are privacy, no subscription, and offline-capable local processing, but it provides largely verbatim transcripts, lacks custom vocabulary, can struggle more with accents or noisy environments than hosted models, and cannot switch languages within a sentence. The comparison frames alternatives around the tradeoff between privacy and polished output: local tools such as Handy and FluidVoice use on-device models and retain audio locally, while cloud or hybrid tools such as Blurt, Superwhisper, and MacWhisper may offer stronger recognition, cleanup, formatting, custom terms, and language features but can require payment or send audio to servers. It also notes that developers can integrate speech recognition through an API rather than using a standalone app, and concludes that the appropriate choice depends chiefly on whether a user prioritizes local processing, improved transcription quality, automated editing, or product integration.

Open-source dictation alternatives for Mac have expanded beyond basic transcription, with the main choice now being whether audio stays on-device or is sent to cloud services for potentially stronger recognition and AI-assisted cleanup. Local options such as Handy, FluidVoice, TypeWhisper, and VoiceInk emphasize privacy, offline use, and no per-hour processing costs, though their accuracy and speed depend on local hardware and models such as Whisper or Parakeet. Cloud-based Blurt uses AssemblyAI’s API to generate cleaned-up, context-aware text that removes filler words and resolves corrections, but requires an internet connection, an API key, and payment of about $0.62 per audio hour. OpenWhispr provides a hybrid approach, allowing users to switch between local and hosted models while learning vocabulary from corrections. FluidVoice and TypeWhisper offer especially broad model support, formatting tools, voice commands, dictionaries, integrations, and customization, while VoiceInk provides locally processed transcription through a one-time purchase. Across the category, developers are increasingly focused on transforming spoken language into polished, appropriately formatted writing rather than delivering literal transcripts, while privacy, platform compatibility, licensing, cost, and configuration complexity remain important selection factors.

Wispr Flow is presented as a capable cloud-based dictation tool whose limits include a restricted free tier, subscription cost, remote transcription, and closed-source design, leading users to seek alternatives based mainly on platform, privacy, price, and openness. For macOS users, Blurt is identified as an MIT-licensed cloud option with transcription cleanup, while VoiceInk offers on-device transcription with a one-time purchase model, and FreeFlow supports user-selected cloud or local providers. Handy is recommended as a free, fully offline, cross-platform option, while OpenWhispr supports macOS, Windows, and Linux with either local models or hosted providers; FluidVoice and TypeWhisper provide additional open-source choices with more limited platform support. Superwhisper is described as the closest commercial replacement, supporting desktop and mobile platforms with local and cloud processing options, whereas MacWhisper focuses more on file and meeting transcription. The comparison emphasizes that mobile and Linux choices remain limited, that MIT and GPLv3 licenses have different implications for commercial reuse, and that developers can alternatively integrate dictation directly through an API that combines speech transcription with automated text cleanup.

AssemblyAI has made its Universal-3.5 Pro speech-to-text model available through OpenRouter’s transcription endpoint, allowing developers to access it with an existing OpenRouter API key, billing relationship, and OpenAI-compatible request format. The Sync Speech-to-Text API is designed for short, turn-based audio interactions such as voice agents, dictation, phone trees, and push-to-talk messages, returning a complete transcript in a single request without polling or persistent WebSocket connections. The service supports WAV clips from 80 milliseconds to two minutes and up to 40 MB, with direct API median latency reported at about 134 milliseconds plus OpenRouter routing time. Developers can improve recognition for specialized or contextual audio through prompts, per-request key terms, and conversation history, while word timestamps are available through OpenRouter’s verbose JSON response options. AssemblyAI states that Universal-3.5 Pro achieves a 1.59% normalized word error rate on short-form audio and supports 19 native languages, with particular emphasis on recognizing proper nouns, alphanumeric strings, email addresses, and postal addresses. Pricing is listed at $0.45 per hour of audio, passed through by OpenRouter without markup, with an introductory first-week discount of 50%.

AssemblyAI’s Sync and Dictation APIs use the same Universal-3.5 Pro speech-to-text model, accept up to 120 seconds of WAV or raw 16-bit PCM audio, and support pre-warming and audio upload during recording, but differ in their outputs and intended applications. Sync returns only a verbatim transcript, making it better suited to regulated records, spoken-punctuation workflows, word-level timestamping, voice-agent turns with conversational context, custom downstream cleanup, and cost-sensitive deployments at $0.45 per audio hour. Dictation costs $0.62 per hour and adds an LLM-generated cleaned version alongside the unchanged transcript, removing filler, resolving self-corrections, and adding formatting for text intended to be sent directly, while allowing customizable output instructions and falling back to the raw transcript if cleanup times out. The cleanup layer improves readability rather than transcription accuracy, so specialized names, terms, and jargon should be supplied through key-term prompting, while contextual prompts can describe the expected audio domain. Dictation necessarily adds some latency because it performs a second processing pass, although both APIs typically process short clips in under a second and expose timing metrics. For recordings longer than two minutes, such as extended clinical notes, the recommended approach is chunking or streaming transcription followed by cleanup of the assembled text.

Dictation APIs should be evaluated differently from conventional speech-to-text services because they handle short, single-speaker, user-triggered clips where users expect immediately usable text rather than raw transcripts. The proposed evaluation framework prioritizes cleanup quality, end-to-end latency, domain-specific jargon accuracy, integration effort, and compliance or data-flow requirements, recommending that teams test vendors using their own messy utterances, vocabulary-heavy samples, production-like latency measurements, limited integration prototypes, and written security documentation. It distinguishes transcription accuracy from rewrite-based cleanup, noting that fluent formatting cannot fix words the speech model misheard, and argues that missed-entity rates may be more meaningful than aggregate word-error rates for specialized workflows. The comparison reviews offerings from AssemblyAI, Deepgram, ElevenLabs, Gladia, Google Cloud, AWS, and OpenAI, emphasizing differences in built-in cleanup, output steering, synchronous delivery, custom vocabulary controls, clip limits, pricing, and HIPAA-related BAA availability. It also advises choosing streaming transcription for live partial results, standard transcription for lengthy or multi-speaker recordings and strict verbatim use cases, custom transcription-plus-LLM pipelines when cleanup is a core product differentiator, and desktop dictation applications when no product integration is needed.

Choosing between a bundled dictation API and a speech-to-text API paired with an LLM depends primarily on whether transcript cleanup is a product differentiator or supporting infrastructure. Building a two-stage pipeline gives teams control over prompts, rewrite models, formatting, and evaluation, but also requires them to manage audio capture, prompt iteration, latency, error handling, vendor relationships, rate limits, and variable token-based costs. The article argues that sequential transcription and LLM calls can produce roughly two-second delays, while a bundled API can return both verbatim and cleaned text in a single request, typically in under a second for short clips. Its cited pricing places bundled dictation at $0.62 per audio hour versus $0.45 for transcription alone, with the difference representing integrated cleanup and more predictable billing. Building remains appropriate for products with specialized rewrite behavior, mandated models, established LLM infrastructure, or long-form and multi-speaker audio needs, while bundled dictation is presented as better suited to short, single-speaker interactions where fast finished text and simpler operations matter most.

AssemblyAI has launched its Dictation API, a speech-to-text service designed to turn spoken utterances into ready-to-send text rather than verbatim transcripts by removing filler words, resolving self-corrections, preserving tone, and optionally formatting output for contexts such as task lists, messages, or clinical notes. Built on the company’s Universal-3.5 Pro model, the API supports 19 languages, costs $0.62 per hour of audio, returns both the original transcript and rewritten text, and allows developers to improve recognition of specialized vocabulary through contextual prompts and key-term spelling biases. AssemblyAI reports typical short-request response times below one second and says audio can be uploaded in chunks while a user is speaking; if output processing times out, the service returns the verbatim transcript instead. The company cites a 3.87% normalized word error rate on short-form English benchmark audio, offers a Python SDK and HTTP integration, and has also released Blurt, a free open-source macOS dictation application that demonstrates the API in use.

Clinical dictation is presented as a short, single-speaker transcription workload distinct from ambient scribing, which captures longer multi-speaker encounters in real time. AssemblyAI recommends the Sync API when applications require a verbatim record of speech and the Dictation API when they need both a faithful transcript and a configurable cleaned or structured output, such as a SOAP note or intake-form JSON. Neither short-clip endpoint supports Medical Mode, which provides enhanced medical-entity recognition on pre-recorded and real-time transcription endpoints, so developers should use contextual prompts and tightly scoped specialty-specific keyterms to improve recognition of drug names, dosages, and clinical vocabulary. The guidance also emphasizes that formatting cannot fix transcription errors, advises testing prompts and keyterms carefully to avoid overcorrection, and notes that prior-visit context can improve medical-term recognition. For responsive user experiences, applications can pre-warm shared HTTP connections before recording ends, while PHI deployments should select appropriate US or EU residency endpoints, maintain endpoint consistency, sign a Business Associate Addendum, and consider downstream redaction and data-storage practices.

Dictation APIs differ from conventional speech-to-text services by producing polished, paste-ready text that removes filler words and false starts, applies punctuation and capitalization, and may allow output instructions, whereas transcription APIs prioritize verbatim fidelity. The comparison recommends evaluating products by time to final text, where cleanup occurs, per-request customization, recognition of specialized vocabulary, language support across transcription and formatting, and operational limits such as clip duration and request format, while treating streaming partials as less important for typical dictation. AssemblyAI’s Dictation API is presented as an integrated, synchronous option that returns both raw and cleaned text, supports configurable rewrite instructions and 19 transcription languages, while Deepgram focuses on spoken punctuation commands, Gladia provides asynchronous audio-to-LLM workflows, ElevenLabs emphasizes low-latency multilingual streaming transcripts, and OpenAI, Google Cloud, Azure, and the Web Speech API generally require separate cleanup or offer tradeoffs in infrastructure, language coverage, and browser support. The discussion argues that building a two-stage transcription-and-LLM pipeline can add latency, maintenance, security reviews, and failure points, and stresses that formatting cannot fix recognition errors in critical terms such as drug names or identifiers, making contextual vocabulary and recognition prompting important.

A New York meetup hosted by AssemblyAI and LiveKit highlighted how Boardy, a networking agent, and Flagler Health, a clinical calling agent, encounter similar production challenges despite serving very different users and operating models. Panelists emphasized that conversational freedom should match the task, with Boardy encouraging open-ended discussion while Flagler tightly constrains calls and escalates medical questions to humans. Both teams favor immediate AI disclosure to build trust, while reliable caller identification and voicemail handling are essential for outbound telephony. The discussion identified multi-party turn-taking, low latency, interrupted instructions, and the gap between what an agent says and what a listener actually hears as major technical issues. Evaluation remains difficult because automated measures can misjudge failed calls and simulated agents do not fully represent human behavior, particularly in regulated healthcare environments with HIPAA requirements. Boardy’s memory design combines compact user profiles with semantic retrieval of past conversations, but voice interactions require this retrieval to happen without creating awkward silence. Across all these challenges, accurate transcription was presented as a foundational dependency for collecting information, evaluating outcomes, managing conversation flow, and retrieving useful context.

Voice AI observability extends beyond traditional telephony health monitoring by making it possible to explain what happened in each live conversation, including what callers said, how speech was transcribed, how the agent responded, where delays occurred, and whether the caller achieved their goal. It should cover transport health, pipeline performance across speech-to-text, language models, text-to-speech, and tool calls, and conversation outcomes such as task completion, escalations, barge-ins, and repeat requests. The article recommends tracking latency at each processing hop with percentile measures, using live-traffic accuracy proxies such as transcription confidence and entity capture, measuring cost per completed task, and retaining transcripts because transcription errors can silently cause otherwise capable models to act on incorrect information. It identifies caller repeat rate as a particularly valuable, low-cost signal of real-world speech recognition failure and advises segmenting it by the preceding agent prompt to locate specific problem areas. Teams can often derive useful metrics from existing session timelines, while adding session IDs, model versions, streaming modes, backend tool spans, detailed error frames, and close codes enables reliable diagnosis. Finally, production failures should be turned into regression-test cases, connecting observability with pre-launch testing and continuous improvement.

Voice-agent load testing should simulate production conditions by running many concurrent end-to-end conversations across speech recognition, language models, speech synthesis, tools, and telephony while measuring accuracy, latency, turn-taking, connection reliability, and cost. It distinguishes concurrency, session-start throughput, and long-duration stability, recommending steady-state, gradual ramp, sudden spike, and multi-hour soak tests because each exposes different bottlenecks such as queueing, rate limits, backpressure, memory leaks, and unclosed sessions. Testing should rely on difficult real-world telephony audio, including compression, noise, accents, interruptions, and entity-heavy information such as account numbers, while auditing transcription ground truth and prioritizing entity accuracy over aggregate word error rate. Results should use latency percentiles, particularly P95 and P99, separated between ramp and sustained periods, and should verify the actual peak concurrency reached rather than the configured target. The text also emphasizes correct audio pacing and encoding, explicit session termination to avoid misleading results and unnecessary billing, provider-limit planning, load generation through simulations, replayed calls, or SIP testing, and a pre-established rollback plan with pinned models, tested fallbacks, retry rules, and measurable thresholds for intervention.

Modern dictation is presented as a broader system rather than simple speech transcription, with users expecting AI cleanup, domain-specific vocabulary support, push-to-talk and hands-free controls, spoken commands and punctuation, contextual formatting, searchable history, and predictable low latency. Cleanup removes filler words and false starts after recognition, while accurate handling of technical, medical, or internal terminology must be addressed earlier through recognition-layer prompting and keyterms, since rewriting cannot repair words that were misheard. Teams commonly use either a bounded-clip approach, in which a user starts and stops recording for a completed transcription, or pause-based chunking for longer dictation sessions such as clinical notes, which provides progressive text but requires carrying context between segments. The piece argues that implementation challenges extend beyond transcription to rewrite-prompt design, error handling, privacy and retention decisions, shortcut configuration, and maintaining responsive performance, with the most important user-facing measure being whether text appears promptly when the speaker finishes.

Dictation systems typically involve two distinct stages: speech recognition, which produces a verbatim transcript, and a subsequent rewrite pass that removes disfluencies such as filler words, false starts, repeats, and some corrections to create readable text. Because rewriting can only polish the words recognized in the first stage, it cannot repair misheard names, technical terms, medications, or other entities; those errors must instead be addressed upstream through domain context, keyterms, conversation history, or specialized recognition modes. The discussion emphasizes conservative cleanup instructions, since overly aggressive rewriting can remove meaningful content or create polished but inaccurate output, and recommends planning for rewrite failures by retaining usable raw transcripts. Verbatim transcription remains preferable for legal, compliance, and certain medical workflows where exact wording, spoken punctuation, or hesitation patterns may matter. It also notes tradeoffs such as custom contextual prompts potentially preventing explicit language selection, and argues that evaluation should prioritize entity accuracy and clearly specified rules about what cleanup must never alter rather than broad word-error metrics alone.

Voice dictation can improve interactions with coding agents such as Claude Code, Cursor, and Copilot by allowing developers to speak detailed, disposable prompts rather than type them, often supplying constraints they might otherwise omit. However, generic speech-to-text performs poorly for developer workflows because it can mistranscribe identifiers and package names, turn syntax into ambiguous prose, retain false starts and filler as contradictory instructions, and lack editor or application context needed to interpret requests correctly. The proposed approach prioritizes transcription accuracy through contextual prompts, project-specific keyterms, and capable speech-recognition models, followed by destination-specific rewriting for terminals, commit messages, or pull requests while preserving the original transcript. It also recommends passing surrounding cursor text, selections, and target-field information as context, using push-to-talk, connection pre-warming, and single-request synchronous transcription to improve responsiveness and reliability. The broader goal is to move beyond dictation as text entry toward voice as executable command input, where a spoken instruction can direct an agent to perform development tasks safely and accurately.