Build a raw WebSocket voice agent: streaming STT, LLM and TTS with no SDK

A detailed Python tutorial demonstrates how to build a voice agent without SDKs by directly managing AssemblyAI’s speech-to-text and text-to-speech WebSocket protocols alongside an OpenAI-compatible LLM Gateway HTTP request. Microphone audio is streamed as 16 kHz PCM to Universal-3.6 Pro Realtime, finalized transcripts are sent with conversation history and weather-tool definitions to an LLM, and generated replies are converted to 24 kHz PCM speech through a persistent TTS socket. It explains the purpose and handling of protocol frames including session initialization, partial and final turns, audio, flushing, cancellation, keep-alives, termination, and errors. The implementation supports barge-in by detecting substantive user speech while the agent is talking, clearing locally buffered playback, cancelling TTS generation, and discarding delayed audio frames until cancellation is confirmed, while filtering short backchannel utterances such as “mhm.” It also covers asynchronous microphone and speaker buffering, tool-call loops, clean shutdown, authentication using one API key, common connection and audio-format problems, rate and session limits, and options for tuning voices, language, transcription context, response latency, and model selection.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.