Build an Agora voice agent with AssemblyAI speech-to-text and text-to-speech

Agora and AssemblyAI can be combined to build a server-side voice agent that joins an Agora RTC channel as a participant, receives a caller’s low-latency WebRTC audio, transcribes it with AssemblyAI’s Universal-3.6 Pro Realtime, generates responses through LLM Gateway, and speaks replies using AssemblyAI Text-to-Speech. The tutorial provides a Python implementation using Agora’s Server Gateway SDK and maintains 16 kHz, 16-bit mono PCM throughout the pipeline, avoiding audio resampling. It explains concurrent audio, transcription, and conversation loops; built-in turn detection; interruption handling that cancels synthesis and clears buffered Agora audio when a caller speaks over the bot; and filtering of short backchannels such as “mhm” to prevent unnecessary interruptions. Setup requires Python 3.10 or later, Agora project credentials, an AssemblyAI API key, and Linux or macOS, with Windows supported through WSL2 or containers. The guide also covers changing voices and languages, improving recognition of domain terms, adding LLM tool calls, reducing response latency through streaming, troubleshooting common credential and audio-format problems, and limitations such as support for one active caller per bot instance.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.