Cost-Effective Logging at Scale: ShareChat’s Journey to WarpStream

ShareChat, an India-based social media platform, transitioned from open-source Kafka to WarpStream to manage their highly elastic workloads and reduce costs associated with inter-AZ networking. This shift enabled ShareChat to save up to 60% compared to multi-AZ Kafka implementations, thanks to WarpStream's auto-scaling capabilities and zone-aware architecture. The platform's machine learning pipeline, which processes logs at ten times the volume of application logs, benefited from WarpStream's stateless, diskless design, alleviating issues like partition rebalancing and leader election inherent in stateful systems like Kafka. ShareChat's implementation of WarpStream involved utilizing Kubernetes for cluster management, leveraging specific agent roles for efficiency, and optimizing batch processing to minimize S3-related costs. Despite slightly increased latency, WarpStream's infrastructure proved effective for ShareChat's machine learning tasks, which prioritize cost savings and operational simplicity over low latency. The transition also included strategic client and Spark optimizations to further enhance performance and cost-efficiency, underscoring the importance of understanding specific workload requirements and trade-offs in latency and infrastructure costs.