March 2026 Summaries

Kaxil Naik, an engineering leader at Astronomer, explains the complexities of managing Apache Airflow's rapid updates, particularly for Astronomer's curated distribution, Astro Runtime, which integrates custom components and heavily relies on Airflow's internal interfaces. Due to the high frequency of upstream changes, Astronomer developed an AI-driven system to detect breaking changes by analyzing daily Airflow commits and classifying them based on their potential impact, using a pattern library derived from past incidents. This system, designed as an Airflow DAG, separates commits into different categories for analysis, allowing engineers to focus on genuine issues and suggest fixes before they affect customers. The AI agent has significantly reduced the time between identifying and resolving potential disruptions, enhancing the reliability of Astro Runtime and contributing fixes back to the open-source community, thereby benefiting all Airflow users.

DoorDash's engineering team addressed the challenges of scaling their Apache Airflow deployment by developing the Orchestration Frederator, a centralized unifying layer that enables horizontal scaling across multiple Airflow instances. This solution was necessitated by operational bottlenecks experienced with a single monolithic instance, which proved difficult to manage at high scale due to memory pressure, DAG parsing delays, and API server responsiveness issues. By categorizing pipelines based on their business importance, DoorDash deployed a tiered instance structure that improved scalability, reliability, and isolation, although it introduced complexities such as managing cross-instance DAG dependencies and dynamically shifting workloads. The Frederator's centralized database and unified interface streamline operations by directing users to the correct Airflow instance and managing dependencies, while the dual-hosting migration strategy ensures smooth transitions. DoorDash's approach highlights the intricate engineering required to scale Airflow, whereas Astro offers a managed service with similar benefits out-of-the-box, including elastic auto-scaling, high availability, disaster recovery, and multi-cloud deployment, allowing teams to focus on pipeline creation rather than infrastructure management.

At the Airflow Summit, Shoubhik Bose from Red Hat detailed how the company updated its data and AI platform using Apache Airflow on the Astro orchestration platform to streamline its complex internal infrastructure. Red Hat, traditionally a software company, faced challenges due to outdated data systems that could not keep up with its rapid growth and hybrid cloud integration. To address these issues, Red Hat developed the Dataverse platform, treating it like an open-source project with a data mesh architecture centered around tools like Snowflake, dbt, and Fivetran, while using Astro for orchestration. This approach allowed Red Hat to deliver real-time, reliable data to executives and support AI agents, such as a business analytics agent that enables natural language queries and a Privacy Impact Assessment agent for regulated industries. The platform's success relies on its integration with Airflow, dbt, and Astro Observe, which ensures data transformation reliability, orchestration across different vendors, and observability for non-technical stakeholders. By leveraging Airflow's capabilities, Red Hat has created a scalable and compliant data and AI infrastructure, enabling efficient data handling and paving the way for future AI developments.

The Astro CLI introduces enhancements to streamline the development and deployment of Airflow pipelines, featuring faster local development through standalone mode and automatic port management, which eliminates the need for container runtimes and simplifies port allocation. The release also includes direct API access via the new astro api command, enabling structured interaction with both the Astro API and the Airflow REST API directly from the CLI. These improvements aim to enhance the efficiency and accessibility of the Astro CLI for developers and AI coding agents, allowing for seamless endpoint discovery and interaction without manual translation. As AI agents become integral to data engineering workflows, Astro is positioning itself as a supportive platform for modern teams, facilitating both local development and autonomous task execution by agents.

At the Airflow Summit, Kunal Jain from American Express presented a session on tackling metadata management at scale using Apache Airflow, focusing on building a scalable metadata pipeline to handle thousands of databases and data sources. He emphasized the importance of metadata in data management and governance, identifying four types: technical, declared, operational, and monitoring metadata. American Express faced challenges in coordinating metadata across a vast landscape of data sources, including RDBMS, NoSQL, and data warehouses, with Airflow serving as the backbone of their solution. The company developed custom Airflow operators to streamline metadata collection, ensuring continuous and reliable updates across their enterprise. Looking ahead, they plan to leverage Airflow 3's event-driven scheduling and remote execution capabilities to enhance their metadata management processes further.

Astro has launched a public preview of its cross-region Disaster Recovery (DR) feature for AWS, marking it as the first managed Airflow platform to offer built-in, one-click cross-region failover. This capability is designed to enhance the resilience of business-critical data pipelines by meeting strict information security, compliance, and enterprise reliability standards. Astro's new feature allows organizations to easily enable DR without the need for custom architecture, extensive engineering efforts, or complex runbooks, addressing the vulnerabilities exposed by regional outages like the significant DNS failure in AWS’s US-East-1 region. The DR feature continuously replicates metadata and task logs to a secondary AWS region, ensuring rapid recovery with a Recovery Time Objective (RTO) of under one hour and a Recovery Point Objective (RPO) of under 15 minutes, validated through rigorous benchmarking. It offers a seamless failover and failback experience via the Astro UI, mirroring deployments in real-time across regions, thus minimizing downtime and maintaining business continuity. Astro's unique offering of native DR capability distinguishes it from other managed Airflow services, positioning it as a critical solution for enterprises that require robust disaster recovery measures without the burden of extensive engineering projects.

Reflecting on past experiences with Magic: The Gathering, the author draws parallels between the strategic deck-building of the card game and the meticulous process of ensuring data quality in Airflow pipelines. Emphasizing the importance of data quality for reliable analytics and AI, the article discusses how Airflow's SQL check operators can be utilized to catch specific data anomalies, such as volume spikes, schema drift, and business rule violations, within a data pipeline. The text introduces six SQL operators—each likened to a strategic card in a deck—that perform various data quality checks, from verifying exact values to monitoring temporal changes. The author stresses the necessity of both Dag-level and platform-level checks to prevent the propagation of faulty data, advocating for a layered approach to data validation. The metaphor extends further with a browser-based game, "Data Quality Duel," designed to help users learn about these operators interactively. The narrative closes by underscoring the critical nature of sequencing checks and the decision-making involved in choosing the appropriate level of intervention when a check fails.

Astronomer developed Kepler, a Slackbot and CLI tool, to address the growing need for scalable access to data and contextual insights across different departments like product management, sales, finance, and marketing. Traditional data team operations often involved handling ad hoc requests and providing guidance, which became unsustainable as the company expanded. Kepler was designed to empower individuals to explore data independently by leveraging a rich context layer built on existing warehouse metadata and code, rather than relying on a complex semantic layer. This context layer incorporates schema metadata, enrichments like table popularity and comments, and code to enable efficient semantic searches and informed SQL query generation. Kepler supports iterative analysis and complex data manipulations via a persistent Jupyter kernel, while playbooks capture successful analysis patterns for repeatability and trust-building. The tool has transformed data interactions at Astronomer, encouraging more users to engage with data and deepening conversations with the data team. By open-sourcing the core components, Astronomer aims to help other companies build similar tools, with ongoing enhancements planned to improve playbook management and automate insights delivery.

Chinni Krishna Abburi, a Business Intelligence Engineer at Visa, presented a guide at the Airflow Summit on how Visa transitioned from manual BI processes to an automated system using Apache Airflow, drastically improving efficiency and data trust. Previously, Visa's BI team struggled with a traditional setup that became inadequate as data volumes increased, leading to delayed dashboard refreshes and loss of stakeholder trust. By implementing Airflow, Visa established an end-to-end automated, reliable, and scalable data pipeline that integrates with modern tools like Databricks, AWS, Tableau, and Power BI. This transformation reduced manual intervention by 70%, decreased data refresh times from 24 hours to under 2 hours, and ensured zero missed SLAs over six months, achieving 100% data trust. The session highlighted lessons learned, such as starting small, prioritizing Dag readability, implementing early monitoring, establishing ownership, and treating Dags with software engineering best practices. Airflow has become the backbone of Visa’s BI operations, providing operational confidence and scalable infrastructure.

Datadog's transition to Apache Airflow 3, as presented by engineers Julien Le Dem and Zach Gottesman at the Airflow Summit, highlights the company's strategic shift from using Luigi and a custom orchestrator to adopting Airflow due to its advanced data-aware scheduling capabilities. Faced with the need for sophisticated data lifecycle management and the challenges of manual intervention in partitioned pipelines, Datadog initially considered building their own orchestrator but ultimately embraced Airflow 3's community-led enhancements. The platform's features, such as multi-tenancy support and worker-specific secrets backends, enabled Datadog to implement a robust multi-tenant setup, aligning with their needs for data-aware scheduling and environment isolation. By leveraging Airflow's open-source community and contributing to its ongoing development, Datadog was able to optimize their orchestration processes and maintain consistency across their operations, demonstrating the power of community-driven innovation in achieving technical goals.

A product manager at Astronomer, with a background in data engineering and leadership, shares insights from the first month of onboarding as an "agent-native" PM, emphasizing the integration of AI into daily workflows. By leveraging Claude Code, a personal workflow agent, they created a comprehensive system for organizing and retrieving structured knowledge from meetings and interactions, enhancing productivity and strategic planning. This system includes a git repository with markdown files for people, customers, and daily logs, and uses tools like Granola for transcription and QMD for semantic search. They also developed an interactive org chart with DuckDB integration for better relationship tracking. Despite achieving significant output, the manager aims to balance detailed documentation with efficiency and plans to make the knowledge system collaborative to benefit colleagues. They advocate for experimenting with AI tools to create compounding systems that enhance work efficiency while focusing human efforts where they are most impactful.