Airflow in Action: Migrating 200,000 Pipelines to Airflow 3 at Uber

Sumit Maheshwari, a Tech Lead at Uber, presented at the Airflow Summit about Uber's ambitious migration project, Operation Airlift, which involves transitioning 200,000 pipelines from their custom-built orchestration system, Piper, to Apache Airflow 3. This change is motivated by the limitations and maintenance burdens of Piper, such as the lack of support for dynamic pipelines and event-driven scheduling, as well as reliability issues, which are addressed by Airflow 3's features like isolated workers, DAG versioning, and a modern REST API. Airflow's proven scalability, familiar syntax, and strong open-source community make it an attractive choice for Uber, which is aiming for a complete migration by the second half of 2026. The migration strategy involves using plugins and providers to maintain close alignment with upstream Airflow and incorporates a GitSync-based delivery mechanism for rapid updates. To ease the transition, Uber developed tools to automate the conversion of Piper's pipelines to Airflow DAGs and is integrating their identity and authorization services with Airflow's API. The end goal for Uber is to establish a robust orchestration environment with AI-assisted authoring, comprehensive observability, and governance layers, while initiating the Piper end-of-life process.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.