The Art of Being Lazy(log): Lower latency and Higher Availability With Delayed Sequencing

WarpStream is a data streaming platform compatible with the Apache Kafka® protocol, designed to leverage cloud object storage for cost-efficient and durable data handling, albeit with a tradeoff in latency. The platform has introduced Lightning Topics, which, when used with S3 Express One Zone (S3EOZ), reduce median produce latency to 33ms and p99 to 50ms without additional costs, by decoupling the durability and sequencing processes. This approach, inspired by the LazyLog paper, allows for asynchronous commitment to the Control Plane, resulting in faster acknowledgments to producers albeit at the cost of no returned offsets, idempotent producers, or transactions. To further ensure data reliability, WarpStream employs a 'Slow Path' to guarantee zero data loss by periodically checking and committing journaled files. Additionally, the new Ripcord Mode enables WarpStream Agents to continue processing produce requests even when the Control Plane is unavailable, although it cannot perform operations requiring Control Plane metadata or start new Agents under such conditions. Both Lightning Topics and Ripcord Mode are now generally available for use.