December 2022 Summaries

In 2022, Apache Airflow experienced a significant year of growth and innovation, marked by over 5,000 commits and contributions from more than 2,300 contributors. This progress was underpinned by a consistent release process, including 12 core releases, biweekly updates, and monthly provider releases. Notable advancements included the introduction of data-dependent scheduling with Airflow 2.4, allowing users to define datasets as part of their DAGs and schedule them based on data changes, which facilitated collaboration across data teams. Dynamic task mapping debuted with Airflow 2.3, enabling tasks to be generated dynamically at runtime, and enhanced with Airflow 2.4 and 2.5 for broader support. The Airflow UI saw improvements with the Grid view and Datasets view, enhancing monitoring and organization of DAGs. Astronomer contributed to the ecosystem with the Astro Python SDK, simplifying DAG authoring by abstracting Airflow-specific syntax, and the release of Astronomer Providers, which introduced deferrable operators for increased scalability. The Airflow community's efforts in 2022 laid the foundation for further innovations in 2023, focusing on data-dependent functionality and simplifying the DAG-authoring experience.

CoRise is offering a new course titled "Effective Data Orchestration with Airflow," with scholarships available for five participants, aiming to enhance skills in data engineering, machine learning, or data science. The course, starting on February 6, spans four weeks and will be instructed by Mike Shwe from Astronomer and Henry Weller from MongoDB. It focuses on Airflow 2.5, providing comprehensive training on using this platform to author, schedule, and monitor workflows for efficient and reliable data pipelines. The course aims to equip participants with valuable, in-demand skills in the field.

Astro has introduced a new feature in its CLI that allows for the deployment of only Apache Airflow Directed Acyclic Graphs (DAGs), aiming to enhance productivity for data teams by providing a faster, more reliable, and flexible deployment process. Previously, deploying code on Astro required creating a Docker image that included the entire Airflow environment, leading to inefficiencies and downtime due to the need for restarting all Airflow components. The new feature, now in public preview, enables DAGs to be bundled into a tarball and mounted directly onto Airflow workers and schedulers, reducing deployment time significantly and allowing for separate management of DAGs and environment files. Astro is also developing additional features to further improve the development experience, such as enhanced operational observability and policies for CI/CD processes. This advancement is geared towards teams that require a streamlined and efficient method for managing Airflow DAGs and associated code, addressing past challenges with Astro's deployment model.

Astronomer has enhanced the Airflow development experience by introducing an improved DAG-testing command in the Astro CLI, which has now been integrated into Airflow 2.5. This development originated from Daniel Imberman's work to create a streamlined DAG testing method in the Astro CLI, allowing users to test DAGs under real-world conditions faster and more efficiently. The new command, astro run, enables authors to rapidly parse and execute DAG code, thus identifying and debugging runtime errors without the need for a full local Airflow environment. This advancement reduces development cycles, enhances code reliability, and enables faster deployment to production. While both Airflow CLI and Astro CLI now utilize a local library for testing, the Astro CLI offers unique features, such as a roadmap for future improvements, and is designed to provide a reproducible Airflow development environment. By supporting the entire software development lifecycle, the Astro CLI allows for efficient DAG building, testing, and deployment, contributing to more reliable production environments.

The Airflow user interface provides various views to understand, monitor, and troubleshoot data pipelines, including the DAGs view, Graph view, Grid view, Calendar view, and Browse tab, which offer insights into task instances, dependencies, and run status in real-time. The DAGs view shows a list of all DAGs with recent run status and metadata, while the Graph view visualizes tasks and dependencies for a specific DAG run. The Grid view displays previous runs and task instance outcomes, and the Calendar view shows DAG run states on a calendar. The Browse tab offers additional views, including Task Instances and DAG Dependencies, allowing users to make changes in bulk. These views have undergone significant updates with recent Airflow releases, making it easier to manage and troubleshoot data pipelines.

The Astro Cloud IDE, introduced by Astronomer, enhances the authoring of Apache Airflow pipelines by providing a user-friendly, notebook-inspired interface that allows data practitioners to write data pipelines using Python and SQL. This tool democratizes access to data orchestration, enabling users like data analysts to independently create and manage pipelines without deep Airflow expertise, thus reducing reliance on data engineers. The IDE facilitates rapid prototyping and development, as illustrated by a case where a data analyst quickly constructs and deploys a data pipeline to refresh dashboard data. The platform supports integration with version control systems like GitHub, promoting best practices for code deployment. Future developments include custom cell types to further extend Airflow's functionality, thereby enhancing the tool's flexibility and utility for various data professionals.

Apache Airflow 2.5 introduces several significant enhancements, notably in dynamic task mapping and data-dependent scheduling, while offering substantial performance improvements and bug fixes. The release exemplifies the benefits of a faster release cadence, allowing users quick access to new features and contributors to focus on manageable changes. Highlights of this version include a revamped `airflow dag test` feature, which significantly accelerates testing by eliminating cold start delays and simplifying log retrieval, and the ability to annotate task instances for better operational documentation. Additional improvements enhance user experience with a less cluttered dataset dependency view, auto-refreshing task logs, and customizable UI settings that persist across sessions. This faster release cycle trend, also adopted by projects like Kubernetes and Apache Kafka, underscores the importance of staying updated with the current-stable version to benefit from new features and critical security patches. Astro, a managed service built on Airflow, simplifies the upgrade process, ensuring users have straightforward access to the latest capabilities, including those in Airflow 2.5.