June 2026 Summaries

The configuration guide for LiveKit voice agents offers an in-depth examination of various turn-taking controls, such as turn detection modes, endpointing delays, interruption modes, and VAD thresholds, which are pivotal in creating a natural conversational flow. The guide assumes familiarity with concepts like turn detection and VAD, and focuses on configuring these features, including new advancements such as LiveKit's audio-based turn detector and adaptive interruption handling. It outlines the pipeline architecture, detailing how user audio flows through each configurable stage, and addresses key questions like detecting speech presence, determining when a user has finished speaking, and handling interruptions. The guide elaborates on different turn detection modes—audio models, STT endpointing, realtime LLM, VAD only, and manual—and offers insights into configuring interruption handling and user turn limits to enhance the agent's interaction quality. Additionally, it addresses common issues and provides resolutions, ensuring developers can tailor LiveKit's capabilities to their specific needs, whether using cloud or self-hosted solutions.

Building a versatile restaurant agent using LiveKit allows for efficient management of multiple restaurants by configuring a single generic agent at runtime with job metadata, such as a restaurant ID, to tailor its response to the selected restaurant. This approach is not limited to restaurants and can be applied to various scenarios, such as support lines or booking assistants, where a single deployment can be configured to adapt to different clients, enhancing user experience by customizing responses based on user-specific data like region, language, or preferences. The backend handles dispatching the agent with the correct metadata, while the frontend manages user interaction, sending restaurant IDs to obtain necessary tokens for room access. Developers should be cautious about potential pitfalls like duplicate room names and ensure consistency in dispatch naming. The demo showcases how to achieve a flexible setup, with the complete example available on GitHub for further exploration.

The text discusses the integration of a voice agent with Supabase, highlighting the technical setup and benefits of this combination. It explores the process of creating a voice agent using LiveKit and Supabase, which utilizes Postgres with pgvector for semantic search, Supabase Auth for identity management, and Row Level Security for data isolation. The setup includes various features such as agentic memory using Reciprocal Rank Fusion for combining vector and full-text search, pre-loaded user context for a seamless user experience, and function-tool CRUD operations for backend interactions. The guide outlines the technical architecture, including embedding model setup and API key management, and provides instructions for running both Node.js and Python agents. The text emphasizes Supabase's strengths in providing a secure and efficient backend for voice agents, offering real identity management, data retrieval, and memory storage without needing a separate vector database or embedding service. It also suggests enhancements like upgrading anonymous users to real accounts, adding multi-tenancy, and hardening the embed function for production use.

LiveKit has introduced two new models, LiveKit Turn Detector v1 and v1-mini, aimed at solving the challenging problem of end-of-turn detection in voice AI, which determines whether a user has finished speaking. Unlike traditional models that rely on text transcripts, these models integrate semantic and acoustic signals directly from speech, reducing latency and errors associated with transcription. The v1 model, available at no cost for agents on LiveKit Cloud, demonstrates superior performance across 14 languages compared to other models like Deepgram Flux and ultraVAD, offering low false-cutoff rates and improved response times. Furthermore, LiveKit released eot-bench, an open benchmark suite to standardize the evaluation of end-of-turn detection, promoting transparency and comparability in model performance. By decoupling turn detection from specific speech-to-text providers, LiveKit allows for consistent conversational experiences across various language models and vendors, providing flexibility and maintaining performance regardless of the underlying technologies used.

The guide provides detailed instructions on maintaining a voice agent's memory across a browser refresh, which typically causes a session disconnect and loss of conversational context. It explains the default session behavior designed for telephony, where a "CLIENT_INITIATED" disconnect ends the session, but highlights how this is unsuitable for web and mobile applications that require session persistence. To keep the agent active, the guide advises disabling the "close_on_disconnect" setting and configuring a "departureTimeout" to ensure the room remains open long enough for a user to reconnect. Additionally, it emphasizes the need for persistent memory through databases for recalling user interactions across separate sessions. The text also discusses the importance of handling different participant types, like web users and SIP callers, by adjusting settings to either maintain or terminate sessions based on the nature of the disconnection, ensuring a balance between resource management and user experience.

Choosing between self-hosting LiveKit or using LiveKit Cloud involves weighing control against convenience, as both options provide the same components, APIs, and SDKs with differing operational responsibilities. Self-hosting grants full control over infrastructure but requires managing services, infrastructure, and ongoing operations, which is suitable for teams with robust DevOps processes and predictable traffic patterns. In contrast, LiveKit Cloud offers a managed service that alleviates operational burdens, providing additional features like managed agent hosting, enhanced noise cancellation, and global network capabilities that are not feasible in self-hosted setups. LiveKit Cloud is ideal for teams needing quick deployment, global infrastructure, and advanced telephony capabilities, while self-hosting is preferable for those with specific compliance needs and a preference for fixed costs. The open-source nature of LiveKit ensures flexibility, allowing teams to transition between hosting models as their needs evolve, with both options enabling scalable and reliable voice AI and media routing solutions.

LiveKit's noise cancellation system is designed to enhance audio quality for voice AI, livestreaming, and telephony by reducing background noise and competing speech, which can degrade transcription accuracy and interfere with turn detection. The platform provides different noise cancellation strategies for various applications: server-side processing for voice AI agents, client-side processing for livestreaming, and SIP trunk-level processing for telephony. LiveKit offers two main types of enhanced noise cancellation models: background noise suppression, which removes non-speech noise while preserving all speech, and voice isolation, which emphasizes a primary speaker by suppressing other voices and noise. These models, provided by Krisp and ai-coustics, should not be stacked on the same audio pathway to avoid unexpected results. Echo cancellation, a separate feature, prevents speaker output from looping back into the microphone and can be used alongside noise cancellation. For optimal results, LiveKit recommends applying noise cancellation within the agent for most voice AI applications, using different models based on whether the scenario involves a single speaker or multiple speakers with environmental noise. Pricing for noise cancellation models varies, with voice isolation being metered and background noise suppression included with LiveKit Cloud.

LiveKit Portal is a sophisticated framework designed to facilitate the remote operation and data collection of robots, addressing challenges such as security, latency, and synchronization. It enables teleoperation by seamlessly integrating observation syncing, which combines camera frames and joint states into coherent, real-time snapshots, ensuring clean and aligned training data. This capability is critical for robotics teams who operate robots they cannot physically reach, allowing them to collect teleoperation data and perform remote inference on real hardware. The system supports multiple operators and policies in a flexible, video-call-like environment, allowing easy handoffs between human operators and automated policies. LiveKit Portal provides observability by offering live metrics snapshots that aid in monitoring and optimizing the robot's performance and handling network variations. It is designed to work effectively both over the internet and within localized networks, offering high fidelity data collection through various video codecs. This versatility enables robotics developers to focus on robot functionality rather than the intricacies of network transport, making it a valuable tool for both local and distributed robotic deployments.

NVIDIA's Nemotron 3.5 ASR is a 600 million-parameter streaming speech recognition model capable of transcribing 40 language-locales with remarkable speed and efficiency, suitable for running on a laptop. The model employs a language-ID prompt to direct decoding, allowing a single set of weights to handle multiple languages such as English, Spanish, and Japanese, with a sub-100ms end-of-utterance latency that ensures transcripts keep pace with spoken words. This post explores its applications, particularly its integration in NeMo, OpenAI-compatible servers, and LiveKit voice agents, highlighting its real-time processing capabilities and local execution on devices like CPUs and Apple Silicon, thus eliminating cloud dependency and reducing costs. A key feature is its multilingual teleprompter functionality, where the model's streaming and low-latency attributes enable the script to scroll in sync with the user's voice, further enhanced by a clever matching algorithm that maintains accuracy and responsiveness. This model stands out in the multilingual streaming space for its speed and local execution, making it a unique solution in speech recognition technology.

A voice agent that lacks memory between calls struggles to provide a personalized experience, as it cannot recall previous interactions or user information. MongoDB Atlas offers a solution by housing personalization, retrieval-augmented generation over a knowledge base, and session memory in a single platform. This guide outlines five integration patterns to connect Atlas with a LiveKit voice agent, showcasing how MongoDB's flexible schemas, aggregation pipelines, and vector search facilitate efficient data handling and persistence. The importance of persistent state in voice interactions is highlighted, as tighter latency budgets require quick access to user data without relying on system prompts. LiveKit's hooks and lifecycle callbacks align with MongoDB operations to enhance personalization, knowledge retrieval, and memory across sessions. Integration patterns include using vector search for knowledge retrieval, agentic memory tools for capturing conversation details, identifying users to pre-load context, CRUD operations via function tools, and session persistence upon call completion. The starter kit provided can be cloned for a quick setup, demonstrating how to implement these patterns using MongoDB Atlas alongside LiveKit's infrastructure for a more responsive and memory-equipped voice agent.

LiveKit has released version 1.0.0 of its C++ SDK, designed to enable developers to create real-time audio, video, and data communication for C++ applications on its infrastructure, which already supports telephony, web, mobile apps, and embedded devices. This SDK is particularly beneficial for robotics, autonomous systems, and physical AI that require low-latency communication with cloud intelligence, without the need to wrap C++ autonomy stacks in other languages. The SDK supports Linux, macOS, and Windows across x86-64 & ARM devices like NVIDIA Jetson and Raspberry Pi, and is integrated with CMake for easy project inclusion. Developers can leverage this tool for a range of use cases, including real-time robotics, low-latency AI inference, and high-performance media applications, with support for dedicated hardware encoders on various GPUs and CPUs. LiveKit plans to enhance the SDK with features like session recording and replay, and a ROS2 bridge, to further support robotics and embedded systems in various environments.