Airflow in Action: Expedia’s Multi-Tenant Platform at 200+ Clusters, 14,000+ Pipelines

Expedia Group transitioned from fragmented Apache Airflow deployments to a centralized, platform-engineered multi-tenant model to enhance operational efficiency, security, and compliance. This shift involved leveraging Kubernetes for isolated environments, Backstage templates for rapid pipeline creation, and fully automated CI/CD processes to streamline deployment and reduce errors. The company now operates over 200 isolated Airflow clusters supporting 180 engineering teams, orchestrating 14,000+ pipelines and 1.5 million task executions monthly. The multi-tenant approach balances autonomy and standardization, enabling teams to focus on business logic while maintaining operational consistency. Challenges such as maintaining consistency across decentralized repositories were addressed through a centralized Python library and configuration-as-code. These improvements reduced the time and effort required for cluster management and deployment, showcasing the benefits of combining secure infrastructure with developer empowerment.