Build a real-time voice agent in Python with tool calling

The tutorial explains how to build a real-time Python voice agent using AssemblyAI’s Universal-3.6 Pro Realtime for streaming speech-to-text, LLM Gateway for reasoning and OpenAI-style tool calling, and AssemblyAI Text-to-Speech for spoken responses, all managed through one API key and SDK. It organizes the implementation into three files for microphone and speaker handling, tool definitions, and the central agent loop, which sends completed user turns to an LLM, executes requested Python functions such as weather lookup or memory storage, and converts the resulting response into streamed audio. A central feature is agent_context, which supplies the agent’s previous spoken prompt to the transcription model to improve recognition of short or structured replies such as email addresses, confirmations, and account identifiers. The example also handles barge-in by detecting meaningful speech while audio is playing, filtering out brief backchannels, and cancelling text-to-speech playback so the user can interrupt naturally. The tutorial discusses latency optimizations including small audio chunks, fast LLMs, persistent TTS sessions, and streamed audio playback, while outlining production adaptations for phone calls, web applications, long sessions, logging, PII redaction, and robust tool-call timeout policies.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.