Robinhood Swaps Kafka for WarpStream to Tame Logging Workloads and Costs

Robinhood transitioned from Kafka to WarpStream for their logging workloads, resulting in a 45% cost reduction. This change was motivated by the need for a more elastic and scalable solution to handle their cyclical and often unpredictable data traffic tied to U.S. stock market hours. WarpStream's auto-scaling capabilities, S3-compatible diskless architecture, and ability to eliminate complex networking requirements such as VPC peering and AWS PrivateLink made it a suitable choice. The migration process involved a phased approach to minimize disruptions, with Robinhood splitting their Kafka setup into separate WarpStream clusters for different functions. They implemented horizontal pod auto-scaling and AZ-aware scaling to efficiently manage traffic spikes and reduce latency. The move to WarpStream allowed Robinhood to simplify operations, reduce storage and compute costs, and eliminate the maintenance of the Kafka control plane, contributing significantly to the overall cost savings.