September 2026 Summaries

Agora’s Conversational AI Engine v2.13 adds custom HTTPS tool calling for multimodal large language model agents, allowing realtime models such as Gemini-3.8-Live and OpenAI GPT-Live-1 to access application APIs and external services without requiring a cascading ASR, LLM, and TTS architecture. The release also brings MCP server configuration for MLLM agents into the Agora Agents SDKs, enabling developers to connect existing internal or third-party MCP integrations directly to voice agents. New model options include OpenAI GPT-Live-1 for MLLMs and Smallest AI for speech recognition and text-to-speech in cascading agents. Generated filler words can now use recent conversation history, with configurable message and character limits, although custom filler prompts now replace rather than supplement Agora’s default prompt. Developers upgrading should configure REST API tools under mllm.tools, enable advanced_features.enable_tools, review custom filler prompts, and ensure non-default ASR configurations include asr.params.

An AI Avatar Masterclass featuring Akool, AKA Virtual, and Trulience examines how successful digital humans depend on clear purpose as much as realistic visuals. Akool emphasizes photorealistic, real-time avatars and edge inference for responsive video interactions, while AKA Virtual uses stylized animated characters to establish expectations and support functional or entertainment-focused roles. Trulience provides a modular visual layer that can connect with different language, speech recognition, and voice systems, enabling developers to adapt for privacy, language, and deployment needs. The discussion highlights the importance of conversational pacing, accurate lip sync, emotionally appropriate expressions, and character-specific voice design, alongside flexible architectures that allow models to change over time. Practical applications include multilingual marketing and communications, media production, customer assistance, and browser-based healthcare information services, with the central recommendation that an avatar’s intended job should guide its appearance, behavior, technology, and interaction design.

Agora highlights Gemini 3.8 Flash TTS and Flash-Lite TTS as text-to-speech models designed to give developers greater control over speech delivery, including emotion, pace, character, accent, and changes in tone during a conversation. Flash TTS offers a library of more than 20,000 voices across languages and regions while retaining existing named voices, enabling teams to select voices suited to different audiences and applications. The models are intended to improve voice consistency during extended interactions and support more natural multi-speaker dialogue through realistic pacing, breathing, and turn-taking. Agora suggests these capabilities could support customer-service agents, educational tools, interactive stories, game characters, and regionally localized voice applications, with the goal of making real-time AI voice experiences feel more natural and engaging.

A discussion with AssemblyAI researcher Luka Chkhetiani argues that voice AI performance should be judged by the complete conversational experience rather than transcription speed alone, since turn-to-turn latency also includes endpoint detection, interruption handling, language-model processing, and response playback. Although AssemblyAI has reduced word emission latency to about 150 milliseconds, users may still perceive delays when other stages disrupt conversational rhythm. The piece emphasizes that reliable production speech AI must behave predictably across noisy, overlapping, interrupted, and otherwise imperfect real-world audio conditions, not merely score well on controlled benchmarks. It also calls for more transparent evaluation that reveals model-specific weaknesses and trade-offs among accuracy, cost, latency, infrastructure, and integration complexity. AssemblyAI’s work on agent context, adaptive conversation windows, and speaker revision is presented as a way to improve understanding, while future voice agents may benefit from recognizing and responding to signals such as frustration, confusion, or hesitation.

Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking are positioned for different voice AI conversation needs, with the former emphasizing low-latency responses for direct, predictable tasks such as scheduling, FAQs, call routing, order status, and basic customer service, and the latter supporting deeper reasoning, multi-step decisions, and coordination across multiple external tools. The post notes that perceived voice-agent responsiveness depends not only on model inference but also on turn detection, network transport, tool execution, and audio generation, making delays especially noticeable in spoken interactions. Gemini 3.8 Live can use asynchronous tool calls to continue conversations while background actions such as calendar or order lookups occur, while Extended Thinking can reason through complex cases such as airline rebooking while gathering user preferences and presenting updates. Both models allow applications to add or update context during active sessions, enabling business data to be introduced when relevant. Developers are advised to select models based on the required balance of speed, reasoning, autonomy, and conversational complexity rather than treating one model as universally superior.

GPT-Live-1 is presented as a full-duplex voice AI model designed to make spoken interactions more natural by listening while speaking, allowing users to interrupt, correct, pause, and add details without restarting the conversation. It supports backchannel acknowledgments, such as brief verbal cues that indicate it is listening, and can remain conversational while a separate AI agent performs research, reasoning, or connected-tool tasks such as checking travel options. Applications can provide prior customer history, live background updates, and changing instructions, while distinguishing between information the assistant should know internally and information it should say aloud. The model can summarize long sessions, accommodate typed input alongside speech, initiate greetings, and support custom voices for eligible projects. The post emphasizes that developers remain responsible for reliable tool integrations, accurate recordkeeping, careful context management, and ensuring that interruptions do not accidentally trigger or misrepresent actions, with Agora directing readers to its documentation for integration details.

Live Play Mobile, which has produced approximately 100,000 hours of live mobile entertainment since launching a 24-hour bingo game in 2020, is developing AI avatars based on its experience that player retention depends on personal recognition and emotional connection as much as game mechanics. The company sees AI and human hosts as complementary: AI avatars can operate continuously, localize rapidly, test hosting approaches at scale, and personalize interactions for large audiences, while human hosts provide enduring personality, judgment, and relationship-building skills. Its development focus includes low-latency, expressive voice technology, as mechanical or delayed speech can undermine real-time game engagement, alongside synchronized language, voice, video, and timing systems. Agora’s Conversational AI Engine provides underlying real-time infrastructure, while Live Play Mobile retains control of proprietary game knowledge, player context, memory, hosting rules, and business logic. The company plans to test formats in which AI hosts recognize individual players within large groups, temporarily shift to one-to-one interactions, and potentially take over from human hosts while preserving continuity between shifts.

A discussion with Bluejay co-founder and CTO Faraz Siddiqi examines the challenges of making voice AI agents reliable in real-world settings, where interruptions, accents, noise, network problems, tool failures, and unpredictable requests can undermine polished demos. Bluejay shifted from building restaurant voice agents to creating testing infrastructure after finding that manually validating one agent could require 10 to 12 hours across numerous ordering and reservation scenarios. Its simulated “digital humans” can place parallel calls under varied conditions, while its Replay feature turns production failures into reproducible regression tests. Siddiqi emphasizes that agent evaluation should focus on whether the intended downstream task is completed correctly, rather than isolated measures such as latency or instruction following, since delays can also alter how callers communicate. He also argues that teams should begin with manual testing and customer listening to develop intuition about real failure modes before scaling automation through simulation, observability, heartbeat checks, and regression testing.