June 2025 Summaries

ShareChat, an India-based social media platform, transitioned from open-source Kafka to WarpStream to manage their highly elastic workloads and reduce costs associated with inter-AZ networking. This shift enabled ShareChat to save up to 60% compared to multi-AZ Kafka implementations, thanks to WarpStream's auto-scaling capabilities and zone-aware architecture. The platform's machine learning pipeline, which processes logs at ten times the volume of application logs, benefited from WarpStream's stateless, diskless design, alleviating issues like partition rebalancing and leader election inherent in stateful systems like Kafka. ShareChat's implementation of WarpStream involved utilizing Kubernetes for cluster management, leveraging specific agent roles for efficiency, and optimizing batch processing to minimize S3-related costs. Despite slightly increased latency, WarpStream's infrastructure proved effective for ShareChat's machine learning tasks, which prioritize cost savings and operational simplicity over low latency. The transition also included strategic client and Spark optimizations to further enhance performance and cost-efficiency, underscoring the importance of understanding specific workload requirements and trade-offs in latency and infrastructure costs.

The blog post explores strategies for disaster recovery and inter-region data sharing with Kafka and WarpStream, emphasizing the interconnected nature of these concerns. It discusses the resilience of OSS Kafka and WarpStream against infrastructure and human-induced failures, noting the limitations of each in worst-case scenarios. The post outlines methods for backing up Kafka data, including traditional filesystem backups, copying topic data into object storage, and continuous replication into a secondary cluster, highlighting MirrorMaker 2 and Confluent Cloud Cluster Linking as popular tools for replication. WarpStream's Orbit feature is presented as a tightly integrated solution for continuous replication, offering a seamless transition between clusters while addressing both human and infrastructure disasters. The discussion extends to sharing data across regions through asynchronous replication or WarpStream's Agent Groups, with a focus on cost-effectiveness and latency considerations. The post concludes with a look into achieving true RPO=0 with Active-Active multi-region clusters for critical use cases, emphasizing the complexity and expense of such configurations.