April 2026 Summaries

Voice and chat agents are being increasingly utilized for structured workflows, such as lead qualification and patient intake, with the aim of generating actionable records rather than open-ended conversations. To address the limitations of prompt-only agents in these use cases, LiveKit Agents has introduced guided data collection flows, which can be developed using either Tasks and TaskGroups in the Agents SDKs or through Data collection mode in Agent Builder. Tasks are reusable units that manage specific conversational objectives, while TaskGroups allow for multi-step flows with shared context and backtracking capabilities. Agent Builder, a browser-based tool, enables users to define fields for data collection, automatically generating the necessary TaskGroups without requiring detailed scripting. Both methods focus on achieving a structured output, which is submitted to a designated endpoint at the end of the session. This structured approach ensures that developers can choose between visual design or code-based control while maintaining flexibility to switch between methods.

The guide provides detailed strategies for effectively using Gemini 3.1 Flash TTS, a speech synthesis model in beta, to generate natural and emotionally appropriate speech. It emphasizes the importance of structuring prompts with clear section labels and a "#### TRANSCRIPT" delimiter to prevent the model from reading stage directions aloud. The guide explains that Gemini interprets prompts as context, which requires precise direction to avoid confusion between narrative instructions and actual speech. Key practices include using commas instead of periods to maintain natural prosody, avoiding universal templates for emotional tags, and ensuring prompts begin with a synthesize-speech preamble. The guide also advises against using certain words like "quiet" or "flat" in style notes, as these can lead to monotone outputs, and recommends sticking to documented audio tags for the best results.

Identity verification for inbound SIP callers in LiveKit is structured across three layers: trunk-level filtering, dispatch-rule-level controls, and participant-level identification, each offering increasing granularity. Trunk-level filtering ensures the trustworthiness of the trunk and can restrict caller access to a predefined list of numbers, though caller ID spoofing is possible. Dispatch-rule-level controls route calls to specific rooms and can use PINs and inbound numbers for additional access control, treated as routing hints rather than secure authentication. Once a call is accepted, it becomes a SIP participant, and the agent can access a range of SIP attributes, including phone numbers and custom headers, to identify the caller. However, these attributes are not definitive proof of identity. Therefore, agents should verify caller identity directly during the conversation using methods such as account numbers, shared secrets, or OTPs to ensure secure handling of sensitive information.

When building AI voice agents, developers must choose between a realtime model and a pipeline architecture, each having distinct advantages and trade-offs. Realtime models, which handle audio input and output in a single step, offer benefits like capturing nuances of speech such as tone and emotion, which can enhance conversational naturalness. However, they often lack the customization and control available in pipeline architectures, which use separate models for speech-to-text (STT), language processing (LLM), and text-to-speech (TTS). Pipelines provide greater flexibility in function calling, debugging, and compliance, making them suitable for regulated industries and telephony applications, though they may suffer from higher latency unless optimized. Hybrid models offer a solution by combining strengths from both approaches, allowing developers to tailor their systems based on specific needs and user contexts, ensuring that the architecture chosen aligns with the intended use case and environment.

The text discusses an innovative approach to AI agent orchestration, focusing on separating safety monitoring from conversation handling to improve efficiency and compliance in voice agents. Traditional methods where guardrails were embedded within the agent's system prompt often led to conflicts between maintaining natural conversation and enforcing complex safety rules, resulting in diluted performance and added latency. To address this, the observer pattern is introduced, which involves a background process that runs parallel to the main conversation, using a separate language model to evaluate transcripts for policy violations and inject corrective instructions into the agent's context without interrupting the conversation flow. This method ensures that the front-line agent remains fast and focused, while the observer handles complex evaluations in the background. The text details the setup and implementation of this pattern using LiveKit's Agents framework, including aspects such as real-time transcript monitoring, asynchronous policy evaluation, and context updates, with practical examples and considerations for production environments. This pattern can be adapted to various domains by customizing the violation categories and guardrail instructions to fit specific application needs.

Agent Console is a real-time debugging tool designed to address the complexities of troubleshooting voice agents, which involve multiple components like speech-to-text, model inference, and SIP. Traditional logs fall short in providing insights into issues like awkward pauses or tool call failures, but Agent Console fills this gap by offering a comprehensive view of the entire voice agent session. It allows developers to interact with live sessions, observe audio waveforms, and track events, latency, and participant states, all within a synchronized interface. This tool supports both audio and video tracks, making it versatile for debugging avatar-based agents as well. Agent Console can be integrated into various development workflows, including Builder agents and SDK agents written in Python or Node, and is accessible through the LiveKit Cloud dashboard. It replaces the LiveKit Sandbox as the primary testing surface, offering a deeper level of debugging essential for modern voice systems and promising future integrations with broader observability workflows.

Wake words are critical for activating voice-enabled devices, and livekit-wakeword offers an open-source solution to improve their functionality, addressing issues like outdated codebases and lack of documentation. This library simplifies and speeds up the process of training wake word models, allowing users to create custom phrases for applications such as smart home assistants, robotics, and in-car systems. Livekit-wakeword outperforms openWakeWord in accuracy and efficiency, reducing false positives and detection errors. It generates synthetic training samples and uses a convolutional-attention classifier to create lightweight, fast models compatible with existing frameworks. Part of the LiveKit ecosystem, it ensures seamless integration and minimal latency, enabling hands-free activation across various platforms. The library supports local model training with straightforward installation and setup, and offers backward compatibility with openWakeWord, facilitating easy deployment in existing systems. Future developments aim to optimize performance on embedded microcontrollers, and community involvement is encouraged for further advancements.

LiveKit's new data tracks feature expands its real-time track infrastructure, initially designed for audio and video, to accommodate binary data from sources like sensors and control systems. This allows for applications such as teleoperation, sensor data streaming, and telemetry, with the same low latency and high throughput as media tracks. Data tracks enable efficient bandwidth management through selective forwarding, ensuring only subscribed participants receive the data, which is crucial in poor network conditions. The infrastructure supports features like user timestamps for latency measurement and allows numerous tracks to be published simultaneously due to their lightweight nature. The internal process involves non-blocking frame push, end-to-end encryption, packetization, and selective forwarding to subscribers, maintaining efficient bandwidth usage. Data tracks are supported across various SDKs, including JavaScript, Rust, Python, C++, and Unity, with more support forthcoming, and detailed setup and API information is available in the documentation.

The Talker-Reasoner pattern is an innovative dual-agent architecture for voice AI systems, splitting the task between a fast, conversational "Talker" and a slower, more deliberate "Reasoner." Inspired by Daniel Kahneman's dual-process theory, the pattern addresses the issue of dead air during complex task processing by allowing the Talker to maintain conversational flow while the Reasoner handles intricate tasks in the background. The fast model, optimized for real-time interaction, and the slower, more capable model communicate through a shared state, enabling seamless integration of results into the conversation once the Reasoner completes its task. This pattern is gaining traction in voice AI production settings, as it enhances user experience by keeping conversations natural and responsive, even during long-running background processes. The architecture is distinct from other agent design patterns, as it allows both agents to operate simultaneously, ensuring that the Talker continues to fill conversational space without waiting for the Reasoner, unless accuracy is paramount. The pattern's implementation requires careful orchestration to decide when the Talker should incorporate Reasoner output, balancing conversational flow with data accuracy, and it is currently supported by frameworks like LiveKit, enabling developers to build and deploy these dual-agent systems effectively.