May 2022 Summaries

Apache Airflow® is a popular open-source workflow-scheduling platform, but when used solely for data orchestration, it can be challenging to scale, manage, and maintain due to its lack of essential features such as built-in version control and CI/CD integration. While building a custom Airflow infrastructure might seem appealing to organizations with engineering prowess, it often results in high costs and complexity, diverting skilled talent from core business-focused projects. Fully managed services like Astro offer a more sustainable and pragmatic solution, providing out-of-the-box capabilities such as seamless integration with existing security infrastructures, data lineage collection, and a single control plane for managing deployments, all while maintaining compatibility with open-source standards to avoid vendor lock-in. Managed services are generally more efficient for organizations of all sizes, as they alleviate the burden of maintaining infrastructure and allow organizations to focus on innovation rather than upkeep.

Astronomer Providers, a new addition to the Apache Airflow ecosystem, are designed to enhance the existing provider framework by introducing asynchronous functionality, which is particularly beneficial for long-running tasks. Apache Airflow providers typically consist of hooks, operators, and sensors, and with over 80 available providers containing more than 880 modules, users can easily connect with various external systems. Airflow 2.0 enabled the independent versioning of providers, allowing users to install specific versions without upgrading their entire Airflow deployment. With the introduction of Airflow 2.2, tasks can now be managed asynchronously, freeing up resources by vacating worker slots when tasks are paused. Astronomer Providers, licensed under Apache 2 and compatible with open-source Airflow, aim to further develop the provider ecosystem by leveraging this asynchronous capability, with ongoing support and maintenance from Astronomer.

Apache Airflow 2.3 is a significant release that introduces dynamic task mapping, allowing the scheduler to trigger tasks based on unpredictable input conditions, optimizing parallel processing without the need for custom code. This feature supports various use cases, such as ETL patterns, by dynamically adjusting the number of tasks according to the input, which enhances efficiency and provides better visibility for troubleshooting. The release also includes a new grid view for improved DAG visualization, a LocalKubernetesExecutor for task execution flexibility, and the ability to store connections in JSON format, among other enhancements. Airflow 2.3 maintains backward compatibility with previous 2.x versions, enabling users to benefit immediately from these advancements while eliminating much of the repetitive work previously required by users.