March 2021 Summaries

The Astronomer Registry has been launched as a discovery hub for Airflow providers, facilitating integration between the Airflow community and the broader data ecosystem. Designed to enhance the utility of the Airflow providers framework, the Registry allows developers and companies to publish providers from any public repository, thus enabling easy access to a wide range of tools for creating highly-contextual pipelines. It introduces an intuitive navigation system for users to access crucial information about provider packages and modules, and offers a robust search feature to improve discoverability. A beta program is also underway, collaborating with partners such as Great Expectations and Fivetran to refine the process of creating Airflow Providers. Future plans include expanding the number of providers, introducing a scoring system for modules, and developing a private registry for sharing custom code within teams.

Airflow 2.0 introduced the TaskFlow API, which significantly enhances task management within Directed Acyclic Graphs (DAGs) by explicitly declaring message passing and implicitly managing task dependencies. This functional API allows for the use of XCom Args, enabling abstraction and inference of task dependencies through Python function calls, and includes a task decorator that automatically generates PythonOperator tasks, simplifying variable handling. Additionally, the API supports custom XCom backends, allowing users to store XComs externally and implement personalized serialization methods. The TaskFlow API aims to streamline task creation and management, with future developments expected to expand its capabilities, including additional decorators for other operators. For those new to Apache Airflow® 2.0, the Astronomer CLI and accompanying Quickstart Guide provide an accessible entry point.

In Apache Airflow 2.0, managing secrets—which include variables, connections, and configurations—is crucial for maintaining the security and functionality of your Directed Acyclic Graphs (DAGs). Secrets can be stored using various methods, each with its own advantages and limitations. Environment variables are simple but lack encryption; the Airflow Metastore DB provides some security with Fernet key encryption but may not be suitable for production due to potential security concerns. Alternative secrets backends like HashiCorp Vault, GCP Secret Manager, AWS Secrets Manager, and Azure Key Vault offer secure and flexible options that extend beyond Airflow deployments, allowing for management, rotation, and auditing of secrets. Integrating with cloud-native solutions like GCP Workload Identity and AWS IAM Roles enhances security and ease of management by associating cloud service accounts with Kubernetes service accounts. Best practices suggest using Airflow for its core orchestration capabilities while leveraging more specialized systems for secrets management, ensuring sensitive information is handled securely and efficiently.