September 2022 Summaries

Airflow 2.4 introduces data-driven scheduling, a feature that enhances its ability to manage data dependencies through a new object type called Dataset. This innovation allows Airflow to automatically trigger downstream Directed Acyclic Graphs (DAGs) once the datasets they rely on are updated, simplifying the design of data engineering workflows. At Astronomer, the new feature is poised to significantly streamline operations by enabling a more reliable orchestration of data pipelines through Astro, their managed Airflow service. The data team at Astronomer is actively refactoring its DAGs, such as those for billing data and ecosystem data, to leverage this feature, reducing complexity and eliminating the need for external task sensors. By using Datasets, Astronomer can automate the triggering of tasks based on the completion of upstream processes, thus removing time-based concerns and simplifying the management of DAG dependencies. This transition is expected to greatly simplify the DAG authoring process and enhance the efficiency of their data platform, reducing the need for custom logic to handle timing issues.

Astro CLI, developed by Astronomer, provides a streamlined, open-source solution for installing Apache Airflow® and running data pipelines locally in under five minutes. Designed to simplify the complexities traditionally associated with data orchestration tools, Astro CLI enables users to test Airflow DAGs and tasks efficiently, promote code to the cloud, and collaborate on shared data pipelines. It offers a local development environment with features like astro dev start and astro deploy, facilitating easy deployment and management of Airflow environments. The CLI also supports advanced testing capabilities, including syntax checking and unit tests with pytest, enhancing the ease of pipeline development. Additionally, the integration with the Astro platform offers users the ability to automate cloud deployments and manage environments seamlessly. With a commitment to expanding its functionalities, Astronomer aims to make data orchestration accessible both locally and in the cloud, inviting contributions from the community and offering opportunities to engage with the ongoing development of the tool.

Apache Airflow 2.4 introduces several significant enhancements, with the standout feature being data-driven scheduling enabled by the new Dataset class, which allows for more granular control over task dependencies and the automatic triggering of downstream DAGs based on the successful completion of upstream tasks. This advancement facilitates the breakdown of large, monolithic DAGs into smaller, manageable units, enhancing performance and simplifying maintenance. Additionally, Airflow 2.4 consolidates scheduling parameters into a single schedule parameter, expands dynamic task mapping capabilities, and offers UI improvements for easier navigation and log access. The release also phases out smart sensors in favor of more flexible deferrable operators, promoting asynchronous event-driven operations. While the new datasets feature currently cannot span across separate Airflow deployments, it represents a significant leap in enabling organizations to optimize data pipeline operations and improve data governance.

Laurent Paris, Astronomer's senior vice president of R&D, discusses with Eric Kavanagh on the DMRadio podcast how the integration of observability with data orchestration enhances dataflows' quality and reliability, serving as a crucial step for business process optimization and fostering team collaboration. By leveraging tools like Airflow, observability provides actionable insights that help diagnose root causes of data issues and improve orchestration layers, thus creating a virtuous cycle of enhancement. Observability not only facilitates visualization and understanding of complex data processes but also challenges and corrects flawed assumptions by offering an objective view of dataflows, akin to a Google Maps for data ecosystems. It establishes a common language across teams, breaking down silos and promoting shared responsibility, while also encouraging more strategic decisions about data collection based on business value and service level agreements (SLAs).

Astro, a fully managed data orchestration platform developed by the creators of Apache Airflow and OpenLineage, has achieved compliance with HIPAA and PCI-DSS security standards, enhancing its capabilities in secure data management for industries handling sensitive information. This cloud-native platform optimizes the open-source Airflow experience, enabling faster pipeline development and improved data ecosystem management while minimizing operational risks. HIPAA compliance on Astro follows a shared responsibility model involving public cloud providers, Astronomer, and customers, requiring adherence to a Business Associate Agreement. Meanwhile, PCI-DSS compliance involves meeting 12 stringent requirements to safeguard credit card data, thereby enhancing data protection and organizational reputation. Astro's additional compliance with AICPA SOC 2 and GDPR, alongside its secure Astro Runtime distribution, positions it as a uniquely qualified service for customers with high data security demands in regulated environments.

Astronomer is a data company utilizing metrics to track the growth of Apache Airflow by analyzing development activity, demand, and community population. They monitor development through GitHub data, focusing on commits, pull requests, and issues, and analyze conversations and votes from Airflow mailing lists. Demand for Airflow is gauged through download metrics from PyPI and Docker Hub, indicating a vibrant and growing interest in the platform. Community growth is assessed by tracking individual contributions to the codebase and interactions on mailing lists, alongside monitoring Slack, which hosts over 25,000 registered users. Data is gathered and managed using Airflow DAGs, and Astronomer offers a free ebook to help beginners get started with Apache Airflow.