February 2026 Summaries

Astro's introduction of Dag-level roles offers a solution for managing access to individual Dags within shared Apache Airflow deployments, addressing the challenges faced by platform teams in large organizations. Previously, teams either risked over-permissioning users or created additional deployments to maintain isolation, both resulting in increased costs and complexities. Dag-level access control allows for granular permissions, offering two built-in roles—Dag Viewer and Dag Author—aligning with Apache Airflow 3's permission model. These roles can be applied to individual Dags or grouped using tags, streamlining access management across users, teams, and API tokens. This approach is integrated into the Astro control plane, allowing permissions to be managed alongside existing infrastructure workflows via UI, API, Terraform, and CLI, enhancing governance and compliance with audit logging. By facilitating secure consolidation of workloads, Astro’s Dag-level roles reduce operational overhead and infrastructure costs while maintaining strict access boundaries within shared deployments.

At the Airflow Summit, Ashok Prakash, Senior Principal Engineer at Oracle, discussed the challenges and solutions in building and operating high-scale AI systems, particularly focusing on the role of Apache Airflow in managing these systems on cloud platforms like Kubernetes. He emphasized the importance of separating infrastructure provisioning from orchestration to prevent bottlenecks, using tools like Terraform for platform provisioning, and employing Airflow as a logic control plane to manage GPU-driven workloads effectively. Prakash highlighted that MLOps involves heterogeneous workflows, and Airflow's ability to handle diverse data types, coordinate scalable compute resources, and ensure operational reliability makes it crucial for AI and ML pipelines. He also provided insights into optimizing GPU usage through dynamic workflows, ensuring operational efficiency and maximizing return on investment. The presentation concluded with a focus on production fundamentals, including CI/CD for workflows, modular Dag design, and security practices, underscoring that successful AI scaling requires coordinated management of compute, data, and teams.

Astro has implemented significant architectural changes to its authentication layer to enhance the reliability, resilience, and scalability of its Airflow platform. By introducing dataplane-based forward authentication, Astro eliminates the centralized Auth Proxy, allowing each dataplane to run its own forward-auth service with URI-aware authentication logic. The rollout incorporates backend controls and UI feature flags to manage the transition smoothly, and deployment-scoped API tokens are introduced to ensure operational continuity during control plane outages. The deployment of Direct Access Tokens further decouples workloads from control plane dependencies, providing organization, workspace, and deployment-level access even during outages. These improvements reduce the risk of authentication-related DAG failures, minimize the impact of control plane incidents, and enhance recovery times, addressing customer reliability concerns and optimizing platform efficiency. While most customers do not need to take action, those with specific network configurations may need to update their filters to accommodate new IP ranges.

Expedia Group transitioned from fragmented Apache Airflow deployments to a centralized, platform-engineered multi-tenant model to enhance operational efficiency, security, and compliance. This shift involved leveraging Kubernetes for isolated environments, Backstage templates for rapid pipeline creation, and fully automated CI/CD processes to streamline deployment and reduce errors. The company now operates over 200 isolated Airflow clusters supporting 180 engineering teams, orchestrating 14,000+ pipelines and 1.5 million task executions monthly. The multi-tenant approach balances autonomy and standardization, enabling teams to focus on business logic while maintaining operational consistency. Challenges such as maintaining consistency across decentralized repositories were addressed through a centralized Python library and configuration-as-code. These improvements reduced the time and effort required for cluster management and deployment, showcasing the benefits of combining secure infrastructure with developer empowerment.

Astronomer has released an open-source AI agent tooling designed to enhance AI coding agents with specialized knowledge of Apache Airflow, enabling developers to work with Airflow's intricacies more effectively within their preferred development environments, such as VS Code, Cursor, or Claude Code. This tooling provides modular skills that guide AI agents in creating, testing, and debugging Airflow workflows, analyzing data warehouse schemas, and handling Dag migrations, among other tasks, by leveraging deep contextual understanding of Airflow and data engineering best practices. The tool is designed to integrate seamlessly with existing workflows, reducing friction for developers who prefer local-first development environments, while also supporting collaborative and community-driven enhancements through its open-source nature under the Apache 2.0 license. The initiative aims to make AI-assisted development accessible to a wider range of users by allowing them to utilize expert Airflow assistance without leaving their current tools, ultimately fostering a more productive data engineering ecosystem.

In the evolving landscape of AI, the concept of a "decision tracing context graph" has emerged as a critical asset for organizations, enabling AI systems to understand and learn from the decisions made within a company. This approach not only records the outputs of AI processes but also the reasoning behind human decisions, making it invaluable for refining AI operations. The blog highlights the use of Apache Airflow's human-in-the-loop (HITL) feature, which allows for the integration of human judgment in AI workflows. By using Airflow's HITL operators, businesses can capture and store the context of decisions, such as human approvals or rejections of AI-generated responses, directly in their workflow orchestration. This information is then made accessible to AI agents, enhancing their ability to make informed and context-aware decisions in future interactions. The integration of Slack with Airflow is demonstrated as a practical implementation of this concept, enabling non-technical users to participate in decision-making without needing to navigate complex systems. Overall, this strategy not only improves AI decision-making processes but also ensures that AI agents continuously learn from past decisions, ultimately leading to more accurate and reliable outcomes.