April 2024 Summaries

Airflow 2.9 has been released with over 35 exciting new features, over 70 improvements, and more than 30 bug fixes, building on trends and requests from the community, including enhancements to data-aware scheduling and dynamic task mapping, which are leveraged by over 40% of Airflow users for common use cases like ETL and MLOps. The release introduces advanced options for scheduling DAGs on datasets with conditional expressions, revamped UI for an intuitive dataset experience, object storage support for datasets, custom task instance names, and object storage updates, including scalable and efficient inter-task communication. Additionally, there are numerous user interface improvements, such as a new task duration page, collapsible sections in task logs, and a button to clear only failed tasks. The release is available for free with a 14-day trial of Astro, offering same-day support for all new Airflow releases.

Testing Airflow DAGs is crucial to ensure error-free, reliable, and performant data pipelines. This guide aims to present real-world scenarios and introduce different types of tests that can be used in data pipelines, such as tests to check for DAG-related errors, code functionality, system integration errors, data issues, integration issues, and data quality tests. These tests help alleviate basic programming errors and business errors downstream, promoting a robust development experience. The guide also discusses where to include these tests in a data pipeline, including DAG parse tests, unit tests, data validation tests, and data quality tests, as well as Airflow Cluster Policies to enforce quality standards. By incorporating these tests, data pipelines can adapt and scale with the data ecosystem, ensuring reliability, accuracy, and integrity of data, which powers business decisions and analytics.