November 2026 Summaries

G.711 remains the standard format for many telephone calls, transmitting 8 kHz, 8-bit companded audio as μ-law/PCMU, common in North America and Japan, or A-law/PCMA, common in many other regions. Its limited frequency range removes much of the high-frequency detail that gives modern 24 kHz text-to-speech voices clarity, while unnecessary transcoding, resampling, or improper buffer padding can further introduce noise, clicks, latency, and degraded interruption handling. For phone voice agents, the recommended approach is to request the carrier’s native 8 kHz μ-law or A-law format directly from the TTS provider, send it at the real-time rate of 8,000 bytes per second, and match the negotiated codec used by services such as Twilio, Telnyx, or SIP trunks. The discussion also highlights practical implementation concerns, including 20 ms frame sizing, clearing queued audio during caller barge-in, correct G.711 WAV headers for playback, and the fact that A-law call paths may require direct A-law output for TTS but decoding inbound audio to 8 kHz 16-bit PCM for speech-to-text systems that do not accept A-law.

OpenAI plans to retire tts-1, tts-1-hd, and specified gpt-4o-mini-tts models on January 6, 2027, requiring users of its `/v1/audio/speech` endpoint either to adopt its session-based Realtime API workflow or migrate to another provider. The material presents AssemblyAI’s OpenAI-compatible Text-to-Speech endpoint as a lower-effort alternative: developers can retain the OpenAI SDK and request structure while changing the base URL, API key, and model name to an AssemblyAI language model. It describes mappings for English OpenAI voice names, native voice options for other languages, available output formats and sample rates, pricing of $15 per million input characters, and differences from OpenAI’s service, including unsupported formats, fixed speed, ignored tone instructions, character-based limits and usage reporting, and lack of word timings. It also emphasizes connection reuse, keep-alive configuration, manual handling of rate-limit retries, and compatibility with LiveKit and Pipecat integrations. For real-time voice agents needing streamed text, interruption handling, persistent sessions, or word-level timing data, it recommends moving beyond the compatibility layer to AssemblyAI’s native WebSocket API or framework plugins.

Natural-sounding voice agents depend primarily on writing replies as spoken conversation rather than polished text, because AssemblyAI-TTS-1 derives pacing, tone, pauses, and hesitations from wording and structure instead of SSML-style markup. The guidance recommends soft or mid-thought openings, short fragments, a maximum of one varied filler, false start, stutter, or trailing pause per reply, and brief first responses, while avoiding disfluencies in important numbers, names, compliance statements, or urgent interactions. Punctuation can refine delivery through emphasis, questions, short pauses, cutoffs, and selective capitalization, but bracketed stage directions are spoken aloud and should be removed. Numbers, dates, phone numbers, addresses, emails, and abbreviations should generally be written in conventional forms, with dashes used when digits or letters must be read individually. Developers are encouraged to embed these conversational rules in an LLM system prompt, prohibit Markdown, emoji, brackets, and lists, test phrasing across chosen voices, and stream text sentence by sentence so audio begins quickly and naturally.

Fish Audio provides both a hosted text-to-speech API and downloadable Fish Speech model weights, with models including the production-oriented S2.1 Pro, a free development tier, locally runnable but non-commercial S2-Pro, and the expressive Drama 3. The hosted API supports cURL and Python SDK use, selectable library voices, reusable configuration settings, emotion tags, instant and persistent voice cloning with consent, and HTTP or WebSocket streaming for low-latency applications. Pricing is based on UTF-8 bytes rather than characters, while concurrency limits depend on prepaid spending; users must also account for model-default behavior, limited output formats for telephony, and S1’s planned retirement at the end of 2026. Running S2-Pro locally requires substantial hardware, typically a 24 GB NVIDIA GPU, and is constrained by licenses that restrict the weights to non-commercial use and limit commercial deployment of the code, especially as a hosted service. The comparison with AssemblyAI-TTS-1 presents Fish as better suited to expressive, creator-focused speech applications, while AssemblyAI is positioned for voice agents needing faster streaming, stronger code and address pronunciation, telephony encodings, flexible concurrency, and US or EU processing.

Choosing a real-time TTS API for voice agents depends less on demo voice quality than on integration behavior such as streaming input, interruption handling, word-level timestamps, native 8 kHz G.711 μ-law or A-law output, framework plugins, and concurrency policies. The guide compares AssemblyAI, ElevenLabs, Cartesia, Deepgram, Inworld, and Rime for LiveKit, Pipecat, Twilio Media Streams, and custom WebSocket implementations, finding that all support streaming and telephony formats while differing in plugin maturity, cancellation semantics, pricing, and rate-limit models. It highlights that per-word timing and same-session cancellation help agents manage barge-in accurately, while native telephony encoding avoids additional transcoding. Benchmark results cited from independent and vendor evaluations position AssemblyAI-TTS-1 strongly for entity pronunciation, latency, and word accuracy, although Inworld TTS-2 Flash scores slightly higher on a predicted listener-preference measure and leaderboard positions can change. The guide emphasizes that providers such as ElevenLabs and Deepgram commonly limit concurrent generations, whereas AssemblyAI counts newly created sessions per minute, a distinction that can significantly affect capacity planning for long-running calls.

AssemblyAI’s Text-to-Speech API provides per-token word boundaries that can support synchronized read-along highlighting and live captions, but correct implementation depends on recognizing that timestamps use a continuous session-level audio timeline rather than request-relative time. Each WordBoundaries frame is associated with a request through its flush_id and includes the request’s audio_start_ms, ordered whitespace-split tokens, start and end times, and timing confidence; zero-length entries represent silent tokens and should generally not be highlighted. To avoid drift, especially after multiple requests or interruptions, applications should record the audio player clock when each request’s first buffer is scheduled and calculate playback timing as that request-specific anchor plus start minus audio_start_ms, rather than adding the offset to an already session-relative timestamp. The guide demonstrates enabling boundaries through WebSocket, Python and JavaScript SDKs, and an HTTP streaming endpoint, while showing how browser audio scheduling, token-index mapping, cancellation cleanup, and early-arriving event handling can maintain synchronization. Because timings are delivered on a best-effort basis and may be unavailable for cancelled, empty, unusually long, or failed requests, applications should verify session configuration and allow audio playback to continue without highlights when timing data is absent.

AssemblyAI and Cartesia now offer broader voice-AI stacks spanning text-to-speech, streaming speech-to-text, and agent-related services, making comparisons dependent on the specific layer rather than a single overall winner. AssemblyAI emphasizes pay-as-you-go component services, with AssemblyAI-TTS-1 supporting six languages, Universal-3.6 Pro Realtime supporting 32 languages, and session-per-minute scaling that allows active sessions to remain open without counting against concurrency limits; Cartesia offers Sonic 3.6 TTS in 44 languages, Ink 2 STT in five languages, monthly shared-credit plans, plan-based concurrency caps, and hosted Managed Agents. Published benchmarks show mixed results: AssemblyAI reports stronger TTS entity accuracy and lower first-byte latency in its evaluations, while third-party STT results vary by dataset, with each provider leading on some measures. Cartesia additionally provides dated model snapshots, pronunciation dictionaries, and more vocal-control options, whereas AssemblyAI offers separate per-layer billing and US and EU TTS data zones. The comparison recommends evaluating both providers on an organization’s own scripts, call audio, languages, traffic levels, pricing needs, and compliance requirements, and notes that teams can combine one company’s speech recognition with the other’s voice synthesis through orchestration frameworks such as LiveKit or Pipecat.

Fish Audio’s alternatives are assessed primarily for production text-to-speech and voice-agent use, where its lower listed price of $15 per million UTF-8 bytes may be offset by byte-based billing for accented text, concurrency limits tied to prepaid spending, US-default data handling, a 99% enterprise SLA, and security certifications still in progress. ElevenLabs is presented as a common choice for creators because of its voice features and broad language support, though its listed per-character pricing is higher, low-latency model language coverage is narrower, concurrency depends on subscription plans, and its free tier lacks commercial rights. AssemblyAI-TTS-1 is positioned for voice agents with $15 per million characters, no hard concurrency caps, US or EU processing, voice cloning, streaming controls for barge-in, compliance certifications, and benchmark claims of faster first audio and more accurate reading of codes and addresses than Fish Audio, while lacking Fish’s current emotion-control capability. Cartesia and Inworld offer telephony-oriented output and plan-based concurrency, while open-weight models such as Kokoro and Chatterbox provide self-hosting flexibility but require users to manage infrastructure and operations. The comparison recommends evaluating providers using real agent scripts, regional latency, peak-call behavior, and accuracy on details such as confirmation codes rather than relying only on headline pricing.

Selecting an AI phone agent voice should be treated as a product decision based on callers’ needs, brand identity, language, accent, pace, energy, and performance over real telephony audio rather than on high-quality demo playback alone. The guidance recommends defining a concise voice persona, matching callers’ accents and the voice’s recorded language where possible, and prioritizing clear, mid-range delivery for short conversational turns involving sensitive information, numbers, codes, or bad news. Because phone systems commonly compress audio to 8 kHz using μ-law or A-law, teams should test several shortlisted voices with actual agent scripts in the intended format or on real calls, evaluating intelligibility, tone, naturalness, and listener fatigue. AssemblyAI’s TTS offering is presented as supporting thousands of catalog voices across six languages, configurable voice and language settings, telephony-ready audio formats, and optional consent-based voice cloning for brands associated with a specific speaker. The proposed workflow is to define the persona, filter voices by language, accent, pace, and energy, test four to six candidates using realistic lines, explicitly configure the chosen voice in production, and reassess performance using feedback from live calls.

Voice cloning creates a synthetic version of a real person’s voice from recorded speech, allowing text-to-speech systems to generate new spoken content with that person’s vocal characteristics. Modern systems can produce instant, or zero-shot, clones from short recordings by extracting a speaker representation, while fine-tuned approaches require substantially more audio and training but can capture a broader expressive range. Recording quality, language, tone, microphone placement, and the absence of noise or other speakers strongly affect results, particularly for instant cloning. Applications include customer-service voice agents, consistent brand voices, accessibility tools for people losing their speech, multilingual dubbing, and media production. The text highlights AssemblyAI’s instant cloning offering, which uses consented clips of roughly 10 to 30 seconds and supports several European languages at standard text-to-speech pricing. It also emphasizes that voices are sensitive personal identifiers, making documented consent, clear limits on use, transparency for listeners, data protection, and compliance with evolving laws essential. Cloned voices remain limited by source-recording quality, restricted emotional range in short samples, reduced quality across languages, and imperfect similarity to the original speaker.