How to build a voice agent with AssemblyAI: streaming STT, LLM, and TTS

AssemblyAI’s voice-agent stack combines Universal-3.6 Pro Realtime for streaming speech-to-text and turn detection, an OpenAI-compatible LLM Gateway for reasoning and tool calling, and a streaming Text-to-Speech API, all accessible with one API key while leaving developers responsible for the conversation loop. The guide demonstrates how to connect the services through standard WebSockets and HTTP, stream 16 kHz microphone audio, process completed user turns, query an LLM and external tools, synthesize 24 kHz speech, and handle barge-ins by distinguishing meaningful interruptions from short acknowledgments and cancelling active speech generation. It highlights context updates to improve transcription of short or specialized responses, offers benchmark claims for recognition accuracy and entity handling, and describes usage-based pricing for each service. It also covers multilingual support, security guidance for browser clients, session keep-alives, and alternatives for teams that prefer orchestration frameworks such as LiveKit or Pipecat over direct protocol-level integration.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.