Revamping data consumption at Meesho for modern data needs

Meesho faced challenges in scaling its data infrastructure, prompting a shift from a managed data warehouse to a data lake architecture, utilizing Starburst Enterprise Platform (SEP) and Spark for enhanced performance. Initial deployments of SEP using CloudFormation templates encountered issues with cluster update times and flexibility, leading to a proof-of-concept migration to Kubernetes with SEP Helm charts. This transition aimed to improve fault tolerance and reduce infrastructure costs by leveraging spot instances and employing dynamic scaling strategies. Subsequent design iterations focused on optimizing query execution times, adjusting cluster configurations, and implementing high availability and graceful shutdown processes. The migration to Kubernetes enabled more aggressive scaling, reduced software costs, and maintained low execution times, ultimately achieving a 30% cost reduction and faster deployment times compared to the previous AWS CloudFormation-based setup.