How We Scaled Change Management to 30+ Environments Without a DevOps Team

WarpStream has significantly expanded its control plane infrastructure from four AWS regions to 24 regions across three cloud providers to meet customer demand for proximity and reliability, focusing on aligning failure domains with cloud providers. This expansion led to the development of a sophisticated deployment system that transitioned from a simple Go-based program to a more robust deployment train strategy, and eventually to a fully rewritten system called trains v2, based on async reconciliation loops. The trains v2 architecture allows for more efficient and reliable deployments across multiple regions, incorporating features like automatic pausing outside business hours, targeted infrastructure changes, and automated monitor gating. The new system maintains operational simplicity by leveraging cloud provider primitives and allows for efficient handling of infrastructure changes, reducing manual intervention and enhancing the deployment process's scalability and reliability without needing a dedicated DevOps team.