May 2026 Summaries

Ably AI Transport addresses the common issue of losing session continuity in AI agent interactions due to connection drops, page refreshes, or device switches by storing session states in a resilient channel rather than tying them to a specific connection. This approach allows for seamless reconnection and continuation of AI conversations without the need to re-run inference, providing a more robust user experience. The system supports bi-directional communication channels that enable user interruptions, known as "barge-in," and facilitate organization-side human-in-the-loop (HITL) scenarios where supervisors can join ongoing sessions with full context. Ably AI Transport also enhances multi-agent coordination by allowing specialized agents to publish progress updates independently of a central orchestrator, utilizing Ably LiveObjects to manage user interactions and state changes in real-time. This infrastructure layer improves the reliability and efficiency of AI-driven applications, offering integration options with various large language models (LLMs) and agent frameworks.

WebSockets are considered the optimal protocol for production AI chat due to their bidirectional and persistent nature, which supports live steering and tool calls, unlike Server-Sent Events (SSE), which is unidirectional. However, WebSockets are stateless at the session level, meaning that when a connection closes due to a proxy timeout, page reload, or device switch, all session state is lost. Reconnection logic can re-establish the transport layer quickly but fails to recover any tokens, tool call results, or agent context that were in progress when the connection dropped. The solution to this issue lies in implementing a session layer that persists conversation state with a session ID, allowing reconnection clients to retrieve all missed information. Ably AI Transport offers a durable session layer that addresses these challenges by storing tokens, agent states, and tool call outputs against a session ID, ensuring that reconnection clients receive complete context. This approach overcomes the limitations of infrastructure defaults like AWS ALB and Cloudflare, which can lead to connection drops, and it supports multi-device sessions, offset-based replays, and protocol fallback to maintain continuity across various network conditions.

Ably, a real-time messaging platform, has been adapted to work as a transport layer for the Vercel AI SDK, providing enhanced features such as multi-device and multi-user support, resumable streams, human handoff, interruptions, and history compaction. The integration aimed to expand the capabilities of the Vercel AI SDK, which traditionally uses HTTP streaming for its UI SDK and is limited by single-request single-response assumptions. Ably's pub/sub model allows for a more dynamic message exchange, overcoming issues like lack of multi-user visibility, resumability, and bidirectional communication that HTTP streaming faces. By implementing a custom transport through Ably, developers can bypass these limitations and extend useChat's functionality, albeit with challenges like bypassing the state machine and lifecycle hooks, which Ably addresses with features like the setMessages function. This adaptation demonstrates how Ably's transport can significantly elevate the interactivity and resilience of AI-driven applications on the Vercel AI SDK, offering developers a broader range of options for managing real-time conversations between AI models and users.

AI agent demos often falter in real-world applications due to unreliable connections and user experience issues, rather than model deficiencies. Despite advancements in AI capabilities, the communication experience with agents across devices remains problematic, with common issues such as broken streams, session immobility, and silent agent failures. The problem lies in the delivery layer, where current frameworks are not equipped to handle durable sessions that persist across network drops and device changes. Companies like Ably are addressing this gap by developing infrastructure to support persistent, multi-device connections, enabling a seamless user experience similar to what Uber and Deliveroo achieved through reliable service delivery. This approach allows even smaller teams without extensive engineering resources to offer robust AI services.

@ably/ai-transport` introduces a branching conversation model that allows for non-linear interaction in AI-assisted dialogues, addressing issues like message editing and tool retrying without losing historical context. Unlike traditional linear chats where past interactions can become cluttered or overwritten, this model utilizes a tree structure where each message can create branches, preserved with identifiers like `msgId`, `parentId`, and `forkOf`. This system allows users to compare different iterations of messages side-by-side, facilitating tasks such as customer support, content drafting, and interactive storytelling. The methodology maintains a comprehensive record of conversation paths, enabling seamless navigation and selection of active branches for further interactions. This branching approach, while requiring more storage and presenting some UI complexities, offers significant advantages in maintaining clarity and choice in dynamic conversational environments.

Ably recently introduced AI Transport, a transport layer designed for streaming large language model (LLM) outputs over Ably channels, offering features like resumability and multi-device continuity. The challenge in token streaming is managing the progressive emission of text fragments, which logically belong to a single message but need to be delivered live as they are produced. Ably's solution is a new pub/sub message type called an append, which updates a message by extending its payload incrementally. This approach involves storing each append in two forms: the fragment itself and the full accumulated message, enabling efficient delivery and history storage without fragment clutter. The design decisions include treating conflation as a semantic property to allow automatic concatenation of appends, avoiding user configuration and enhancing efficiency. This method reflects Ably's philosophy of building generic primitives over bespoke features, as demonstrated by their earlier work on chat message editing and deletion, which facilitated the development of AI Transport by extending existing capabilities with minimal additional effort.

AI agent products, though technically capable of performing complex tasks such as holding long conversations and integrating with external tools, often struggle with reliability and user experience due to infrastructural issues unrelated to the models themselves. Challenges arise in maintaining continuous and seamless interactions as sessions frequently break due to network unreliability, device switches, and other disruptions. These issues are not caused by the AI models but rather by the lack of an effective delivery layer that can ensure persistence, multi-device access, and seamless recovery from interruptions. Companies across the industry are independently attempting to solve these problems by building custom solutions or using third-party tools like WebSockets and Redis Streams, but this often requires significant engineering effort. This has led to the emergence of "durable sessions," which offer persistent, addressable connections that maintain session continuity across devices and network failures. The need for a robust delivery layer is increasingly recognized as crucial for ensuring a reliable user experience, allowing AI products to be both effective and user-friendly.

The blog post discusses the challenges of building conversational AI features using standard HTTP streaming and introduces the Ably AI Transport SDK as a solution to these issues. Traditional methods face problems such as losing in-progress streams when a connection drops, lack of multi-device continuity, and difficulties in managing multi-client coordination and concurrent interactions. The Ably AI Transport SDK addresses these by using Ably channels to create a durable session layer that supports resumable streaming and multi-client interactions, abstracting away the complexities of managing concurrent turns and message relationships. This SDK allows developers to build robust AI streaming applications without altering their existing tech stack, using a codec model to bridge different provider and framework event schemas, and is already being used in production with support for platforms like the Vercel AI SDK.