Company
Date Published
Author
Richard Artoul
Word count
2607
Language
English
Hacker News points
8

Summary

WarpStream is an Apache Kafka protocol-compatible data streaming system built directly on-top of object storage. WarpStream offloads all storage to object storage and requires no local disks, making it 5-10x cheaper for high volume workloads. Unlike Apache Kafka, WarpStream has stateless Agents that can serve writes or reads for any topic-partition at any time. To achieve zone-aware routing within the confines of the pre-existing Kafka protocol, WarpStream uses a custom DNS server to generate unique, case-insensitive hostnames for each Agent, allowing clients to distinguish between them even when running behind a load balancer. This enables efficient and scalable data streaming with reduced inter-zone networking costs while maintaining high availability.