August 2025 Summaries

Goldsky, a Web3 developer platform, offers real-time and historical blockchain data access, facilitating dApp development by indexing and streaming data via API endpoints, subgraphs, and streaming pipelines. Initially built on Apache Kafka, Goldsky faced scaling, cost, and reliability issues, prompting a migration to WarpStream. This transition brought significant cost reductions and improved reliability, thanks to WarpStream's diskless architecture, auto-scaling capabilities, and efficient tiered storage implementation. WarpStream's unique architecture decouples hardware from partition counts, allowing for seamless scaling based on workload demands, and eliminates networking costs by aligning traffic zonally. Goldsky's continued growth has been supported by WarpStream's adaptation to accommodate large data clusters, including a redesigned storage engine that optimizes metadata tracking and compaction processes, thereby enabling the platform to handle extensive data storage needs efficiently.

The blog post discusses the integration of WarpStream, a Kafka-compatible streaming platform, with Tigris, a multi-cloud object storage platform, to create a scalable and efficient messaging queue system. WarpStream innovates on traditional Apache Kafka architecture by storing data directly in object storage, eliminating the need for local disks and simplifying operations. This setup prevents costly inter-AZ network traffic and minimizes cross-cloud data transfer fees while maintaining performance. Tigris enhances this by dynamically placing data in the region where it's accessed, offering a globally distributed storage solution without egress fees. The article provides a step-by-step guide on deploying a WarpStream cluster backed by Tigris using Docker, detailing necessary prerequisites, commands, and configurations to get the system operational. This integration ensures data is stored efficiently and remains accessible worldwide, enabling users to create and manage durable message queues without hidden costs or complex infrastructure management.

WarpStream has introduced Multi Region Clusters to enhance the resiliency of its systems, ensuring minimal downtime even if an entire cloud provider region fails. Unlike standard clusters backed by a single control plane region with a 99.99% availability guarantee, Multi Region Clusters utilize multiple control plane regions and a replicated data plane, allowing them to survive the disappearance of a whole region without losing any ingested data or experiencing more than a few seconds of downtime. This architecture employs a quorum of three object storage buckets for writes, ensuring data durability, and uses leader election to manage control plane traffic, reducing latency and write conflicts. The system's design accommodates both hard and soft failures, allowing for seamless leadership transitions and maintaining functionality with no data loss or manual intervention, achieving a Recovery Point Objective (RPO) of zero. Multi Region Clusters are offered with a 99.999% uptime SLA, providing robust protection against regional outages and making WarpStream suitable for mission-critical workloads.