May 2026 Summaries

Organizations' AI initiatives rely on modern data pipelines, prompting many to migrate from outdated orchestration tools like Control-M and AutoSys to Apache Airflow, which supports dynamic, event-driven workflows essential for AI. However, these migrations often stall due to the complexity of converting legacy systems to Airflow's format and the rarity of expertise in both systems. Otto, Astronomer's data engineering agent, addresses this by incorporating Astronomer's extensive knowledge of Airflow best practices, enabling smoother and more efficient migrations. Otto ensures that the converted Dags are optimized for specific environments, retaining institutional knowledge and conventions for future use. It supports migrations from various schedulers and Airflow environments, providing a deterministic and traceable output that aligns with the source job definitions. By storing migration patterns and conventions in Otto Memory, organizations can maintain a context layer for future tasks, enhancing operational efficiency across the data pipeline lifecycle. Otto is now available for early access, offering organizations a structured plan for transitioning to managed Airflow environments like Astro, which alleviates the operational burdens associated with self-managed Airflow.

Bloomberg's Data Platform Engineering team, responsible for managing over 2,500 dynamic Dags to deliver alternative financial data to Bloomberg Terminal subscribers, showcased their innovative use of Apache Airflow configurations at the Airflow Summit. Software engineers Yu Lung Law and Ivan Sayapin explained how they addressed four production challenges without writing new code by fine-tuning native Airflow parameters. These adjustments improved Dag parsing time, resolved a scheduler memory leak, and enhanced their monitoring through StatsD integration. The team emphasized the importance of testing configuration changes in lower environments, using temporary fixes while investigating long-term solutions, and staying updated with the latest Airflow capabilities. They also highlighted the benefits of using Astro, a managed Airflow platform by Astronomer, which simplifies monitoring and observability, allowing teams to focus on data product development with real-time insights and streamlined operations.

Astronomer's Customer Reliability Engineering (CRE) team has been recognized for its exceptional support services, striving to be the top software support team globally, particularly for users of Astro with critical workloads. As the sole managed service provider for Apache Airflow that offers support for Airflow itself, Astronomer faces unique challenges, given the flexibility and complexity of Dags, which can integrate with numerous systems and lead to diverse issues. The team's strength lies in its technical depth, hiring individuals who seek deep understanding and are not deterred by difficult problems, supported by partnerships with R&D and Documentation teams as well as Airflow PMC members. Despite achieving high customer satisfaction and recognition, the team identified the need to incorporate hospitality principles into their support services, inspired by the fine dining industry's approach to service. They aim to enhance customer experience by improving communication, acknowledging their limitations, and providing 24x7 service that feels seamless through better internal coordination and proactive monitoring. Astronomer CRE is committed to developing a superior class of software support driven by technical expertise, a strong feedback loop into product development, and genuine care for customers, recognizing that while they have made progress, there is still work to be done to reach the highest standards of service.

Astro Private Cloud 2.0 is a newly available platform designed for enterprises needing to maintain workflow orchestration within their own environments, addressing concerns related to data sovereignty, regulatory compliance, and infrastructure control. Built on the strengths of Security, Reliability, and Scalability, the release enhances features like Audit Logging, Disaster Recovery, and Config Governance, allowing enterprises to manage large-scale Airflow deployments with improved autonomy and control. Disaster Recovery enables cross-cluster failover with two modes, while Config Governance introduces a hierarchical configuration system for streamlined management. Enhanced security is achieved through detailed Control Plane Audit Logging, capturing structured events across multiple operations and ensuring compliance with various regulatory standards. The platform also facilitates migration from legacy schedulers like Autosys and Informatica, offering modern Python-native pipelines and observability within existing security perimeters. Additionally, operational improvements include direct upgrade paths, strict schema validation, and comprehensive metrics collection, positioning Astro Private Cloud 2.0 as a robust solution for enterprise-scale orchestration needs.

Sumit Maheshwari, a Tech Lead at Uber, presented at the Airflow Summit about Uber's ambitious migration project, Operation Airlift, which involves transitioning 200,000 pipelines from their custom-built orchestration system, Piper, to Apache Airflow 3. This change is motivated by the limitations and maintenance burdens of Piper, such as the lack of support for dynamic pipelines and event-driven scheduling, as well as reliability issues, which are addressed by Airflow 3's features like isolated workers, DAG versioning, and a modern REST API. Airflow's proven scalability, familiar syntax, and strong open-source community make it an attractive choice for Uber, which is aiming for a complete migration by the second half of 2026. The migration strategy involves using plugins and providers to maintain close alignment with upstream Airflow and incorporates a GitSync-based delivery mechanism for rapid updates. To ease the transition, Uber developed tools to automate the conversion of Piper's pipelines to Airflow DAGs and is integrating their identity and authorization services with Airflow's API. The end goal for Uber is to establish a robust orchestration environment with AI-assisted authoring, comprehensive observability, and governance layers, while initiating the Piper end-of-life process.