Engineering message appends for AI Transport: three vignettes

Ably recently introduced AI Transport, a transport layer designed for streaming large language model (LLM) outputs over Ably channels, offering features like resumability and multi-device continuity. The challenge in token streaming is managing the progressive emission of text fragments, which logically belong to a single message but need to be delivered live as they are produced. Ably's solution is a new pub/sub message type called an append, which updates a message by extending its payload incrementally. This approach involves storing each append in two forms: the fragment itself and the full accumulated message, enabling efficient delivery and history storage without fragment clutter. The design decisions include treating conflation as a semantic property to allow automatic concatenation of appends, avoiding user configuration and enhancing efficiency. This method reflects Ably's philosophy of building generic primitives over bespoke features, as demonstrated by their earlier work on chat message editing and deletion, which facilitated the development of AI Transport by extending existing capabilities with minimal additional effort.