April 2026 Summaries

Otto is a data engineering agent designed to alleviate the operational workload of data teams by integrating with Airflow, allowing engineers to focus on more critical tasks. Built by Astronomer, Otto leverages extensive operational Airflow knowledge and provides a range of functionalities, including pipeline building, failure investigation, and upgrade planning, all tailored to an organization's specific environment through its memory-based context. The agent connects to over 25 different data warehouses and offers capabilities such as querying, profiling, and tracing lineage, thus streamlining the data exploration and pipeline management processes. With a foundation rooted in Astronomer's proprietary knowledge, Otto is designed to understand and apply team-specific conventions and corrections, ensuring that institutional knowledge is preserved and accessible. Currently available for Astro customers, Otto aims to advance towards autonomous data engineering, offering proactive diagnostics and multi-step upgrades in the future while continuously expanding its integration capabilities.

Apache Airflow introduced the apache-airflow-providers-common-ai 0.1.0 in April 2026 as an official provider for AI and LLM workflows, enhancing the existing airflow-ai-sdk by integrating features like durable execution, built-in human review, and a new configuration model. This update allows more flexible and manageable AI workflow configurations by using Airflow's connection layer to handle model, API key, and endpoint details, eliminating the need for code changes when switching models or environments. The migration from the SDK to the new provider involves minimal code changes, primarily in dependencies and decorator parameters, and introduces the HookToolset, which converts over 350 existing Airflow hooks into agent tools. This development signifies a commitment to making LLM integration a core feature of Airflow, providing data engineers with a more seamless, robust, and scalable solution for AI orchestration in their workflows.

Cross-region disaster recovery (DR) on Astro is now generally available for AWS data planes, allowing customers to seamlessly fail over their Airflow workloads to a secondary region with just a click. The DR solution was developed to meet business-critical demands from industries like financial services and healthcare, providing an essential backup for enterprise-scale Airflow operations. This innovation alleviates the burden of building parallel infrastructure for DR, which traditionally required significant engineering effort. The system operates by provisioning a secondary EKS cluster in a warm standby mode, ensuring continuity through data replication across three categories: Airflow metadata, task logs, and container images. The architecture relies on AWS Aurora Global Clusters for efficient cross-region replication, bi-directional S3 replication for task logs, and a headless database setup that optimizes costs by running compute instances only when necessary. Programmatic control is available via the Astro API and Terraform, facilitating automated DR operations. Observability and health monitoring are maintained across both primary and secondary clusters, with a focus on centralizing DR awareness in the manifest system to simplify maintenance. Future developments include extending DR support to GCP and Azure, along with enhancing the self-service migration experience for existing clusters.

In the deployment of AI in enterprises, a common issue arises not from the AI models themselves but from the lack of contextual understanding by AI agents, which leads to errors in decision-making due to incomplete or outdated data context. This problem stems from the fact that AI agents lack the nuanced understanding of data context that human analysts possess, such as the history of data production, governance, and operational signals, which are not captured in static metadata or catalogs. The orchestration layer, such as Apache Airflow, plays a crucial role in providing this operational context, as it records the execution history and current state of data, which are essential for making informed decisions. Enterprises that successfully bridge the gap between AI agents and data context do so by integrating orchestration with cataloging systems, ensuring that context is not only documented but also governed, current, and traceable. This approach requires treating context as a form of infrastructure rather than a one-time documentation effort, allowing for continuous updates and improvements. By focusing on building this foundation, organizations can enhance the reliability and trustworthiness of their AI deployments, creating a competitive advantage that extends beyond mere model performance.

Cosmos 1.14 has been released, offering significant enhancements for orchestrating dbt with Apache Airflow, particularly focusing on the Watcher execution mode and comprehensive documentation updates for easier adoption and scaling. The Watcher mode, which has gained traction since its experimental phase, allows for up to 80% reductions in Dag runtime by running a single dbt process per Dag run and utilizing Airflow sensor tasks to monitor model status in real time. This release introduces improved reliability, smarter resource routing, and enhanced visibility for dbt tests, aligning test visibility with model visibility and mapping test outcomes to Airflow task states. The update also includes better debugging capabilities, optimized performance for large Dags, and new features such as task-level memory tracking and organized Dag structures. Cosmos now supports dbt-loom for orchestrating across multiple dbt projects, offering end-to-end visibility. The revamped documentation focuses on practical guidance and common support topics, and Cosmos 1.14 integrates seamlessly with Astro for enhanced observability.

Kaiser Permanente's Division of Research operates complex AI and ML workloads using Apache Airflow on an entirely on-premises, Kubernetes-based infrastructure, emphasizing data privacy by avoiding cloud solutions. Lawrence Gerstley, Director of Data Science, highlights the organization’s reliance on Airflow to streamline compute resource access and facilitate extensive research workflows without requiring researchers to manage Kubernetes directly. This setup supports a wide array of AI and ML applications, including predictive models for medical conditions and text analysis for early disease detection, with Airflow orchestrating tasks across significant computational workloads. While Kaiser Permanente's approach underscores a commitment to maintaining strict control over sensitive health data, it also reflects broader trends in regulated industries prioritizing data residency and sovereignty. The organization is upgrading to Airflow 3 and exploring LLM-assisted Dag authoring to further democratize and enhance their research capabilities. Astronomer offers solutions like Remote Execution and Astro Private Cloud, catering to institutions that need to maintain data within their infrastructure while benefiting from managed orchestration services.

Blueprint in Astro is an innovative tool designed to streamline the process of creating data pipelines by bridging the gap between data analysts and platform engineers, who traditionally encounter bottlenecks due to Airflow's code-first nature. Blueprint allows platform and data engineers to define reusable pipeline templates with a Python class that encodes standard patterns, such as daily ETLs or dbt runs, ensuring consistency across deployments with built-in error handling, connection patterns, and observability. These templates are then used by analysts through a no-code interface in Astro, where they can build and configure pipelines using a drag-and-drop method without needing to write Python or YAML code. The new features in Blueprint v0.2.0 enhance flexibility and maintainability by allowing runtime parameter overrides, providing improved validation error messages, and enabling dynamic configuration using context proxies for Jinja2. This release aims to simplify pipeline creation, offering a seamless integration with existing Git workflows for auditability and governance.

In the evolution of data orchestration, this text illustrates the transformative role of Blueprint, a Python library from Astronomer, which empowers non-experts to create Airflow pipelines through abstraction, breaking down the barrier of technical literacy. Inspired by Ada Lovelace's separation of computational intent from mechanical execution, Blueprint allows platform teams to develop reusable templates in Python while enabling others to configure and compose these templates into pipelines using YAML or Astro IDE's visual interface. This shift addresses common challenges such as lack of Airflow literacy by providing a governed self-service model where quality is derived from well-tested templates rather than individual authoring. Validation, version management, and operational standards are enforced through the schema, ensuring consistency and reducing the bottleneck on engineering teams. The Astro IDE further enhances accessibility by offering a visual builder for crafting data workflows, thus democratizing pipeline creation in a structured and scalable manner, much like Lovelace's vision of a machine that accepts instructions from a broad audience.

Airflow 3.2 builds upon its predecessors by enhancing data-awareness and performance, introducing asset partitions for more granular tracking of data changes, and offering native asynchronous support in the Python operator for more efficient execution of async tasks. This release also improves the Task SDK, API server, and deadline alerts functionality, while unveiling a new provider registry to enhance discoverability within the ecosystem. Users can now customize the Airflow UI theme to differentiate environments, and enjoy better performance in managing large Dags due to grid view virtualization. The update marks a significant step forward in Airflow's evolution, thanks to contributions from its active community, and continues to support scalable and flexible data pipeline management.

Astronomer Academy has developed an AI-driven system to streamline the handling of support tickets related to Airflow certifications, which typically require routine actions such as checking exam statuses and issuing badges. The system employs Apache Airflow to manage the end-to-end ticket lifecycle, using AI agents for tasks that require natural language processing, like categorizing tickets and drafting responses, while deterministic code handles business logic such as making API calls and querying databases. This setup significantly reduces the time spent by the Education team from several minutes per ticket to just seconds of human review, as each response is presented for approval in Slack. This architecture ensures that AI is used where language understanding is needed, while predictable, rule-based tasks are managed by testable, auditable code, maintaining efficiency and accuracy in customer support operations.

At the Airflow Summit, Sagar Sharma from SAP detailed the development of a production-grade Retrieval Augmented Generation (RAG) pipeline using Apache Airflow, which supports Joule for Consultants, SAP's AI copilot. This system processes over 5 million documents from more than 15 data sources, offering a 30% productivity boost and 40% faster ABAP code interpretation for consultants by leveraging SAP-specific knowledge. The team selected Airflow over alternatives like Prefect, Dagster, and Flyte due to its fast implementation, DevOps compatibility, managed service options, and strong community momentum, all aligning with their Python-native infrastructure. The pipeline's evolution involved transitioning from a single hard-coded Directed Acyclic Graph (Dag) to a more flexible architecture with Airflow Variables and separate, parallel pipelines for ETL and data injection, resulting in a scalable system that accommodates both production workloads and AI/ML experimentation. The pipeline includes six modular stages, such as raw data ingestion, preprocessing, chunking, metadata extraction, PII redaction, and vector DB injection, with custom operators tailored for AI-specific tasks.

At the Airflow Summit, Alida Laney, Data Engineer for the City of Pittsburgh, presented how a small two-person team effectively utilizes Apache Airflow and Astro to modernize the city's data infrastructure and enhance operational efficiency across various departments. Faced with unique governmental challenges, including stringent vendor restrictions and the need for inter-departmental data sharing, the team transitioned from Google Cloud Composer to the fully managed Airflow service, Astro, resulting in a significant reduction in infrastructure management time. This change enabled the team to focus on building comprehensive data pipelines that support a wide range of civic needs, including open datasets used by local universities and community organizations, and tools like OneStopPGH Insights for city planning and public transparency. The automation of previously manual processes, such as cross-checking property purchase applicants against permit violations and tax records, exemplifies Astro's impact, reducing a task that once took weeks to mere minutes. This commitment to data-driven governance has earned Pittsburgh a Gold-level What Works Cities Certification, highlighting the city's ability to deliver efficient and reliable services similar to larger cities like Boston and Seattle. Looking ahead, the team plans to expand their data initiatives, including an internal data mart using Google Dataplex, while continuing to democratize data and refine government efficiency through community-driven open datasets.