Airflow in Action: Expedia’s Multi-Tenant Platform at 200+ Clusters, 14,000+ Pipelines

Expedia Group transitioned from fragmented Apache Airflow deployments to a centralized, platform-engineered multi-tenant model to enhance operational efficiency, security, and compliance. This shift involved leveraging Kubernetes for isolated environments, Backstage templates for rapid pipeline creation, and fully automated CI/CD processes to streamline deployment and reduce errors. The company now operates over 200 isolated Airflow clusters supporting 180 engineering teams, orchestrating 14,000+ pipelines and 1.5 million task executions monthly. The multi-tenant approach balances autonomy and standardization, enabling teams to focus on business logic while maintaining operational consistency. Challenges such as maintaining consistency across decentralized repositories were addressed through a centralized Python library and configuration-as-code. These improvements reduced the time and effort required for cluster management and deployment, showcasing the benefits of combining secure infrastructure with developer empowerment.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.