Airflow in Action: Unlocking Airflow 3's Power for Multi-Tenancy at Datadog

Datadog's transition to Apache Airflow 3, as presented by engineers Julien Le Dem and Zach Gottesman at the Airflow Summit, highlights the company's strategic shift from using Luigi and a custom orchestrator to adopting Airflow due to its advanced data-aware scheduling capabilities. Faced with the need for sophisticated data lifecycle management and the challenges of manual intervention in partitioned pipelines, Datadog initially considered building their own orchestrator but ultimately embraced Airflow 3's community-led enhancements. The platform's features, such as multi-tenancy support and worker-specific secrets backends, enabled Datadog to implement a robust multi-tenant setup, aligning with their needs for data-aware scheduling and environment isolation. By leveraging Airflow's open-source community and contributing to its ongoing development, Datadog was able to optimize their orchestration processes and maintain consistency across their operations, demonstrating the power of community-driven innovation in achieving technical goals.