July 2024 Summaries

Astronomer, an Apache Airflow® and dbt™ integration provider, has extended its commercial platform, Astro, to support dbt Core. This move aims to address the operational challenges faced by customers running both Airflow and dbt together. The integration is currently the most popular method for combining these tools, with over 1.3 million downloads per month. Astronomer's commitment to maintaining its open-source Cosmos project while investing in Astro demonstrates its dedication to supporting the broader data ecosystem.

The new release of the Astro Platform introduces features that enhance data orchestration experience by unifying your data estate, improving pipeline resilience, and automating infrastructure provisioning and management. Key features include dbt Deploys for seamless integration with Airflow, Universal Metrics Export for comprehensive monitoring, Self-Healing Workers to ensure system uptime, Astro Terraform Provider for simplified infrastructure management, and Customer Managed Workload Identity on AWS for enhanced security. These improvements aim to empower organizations to achieve greater efficiency, reliability, and scalability in their data orchestration processes.

The Apache Airflow has established itself as the de facto standard for data pipelines due to its flexibility and scalability. However, this has led to a growing need for orchestration tools that can be easily used by different personas, including data analysts, data scientists, and users of legacy tools. To address this, an open source project called DAG Factory was created, which allows DAGs to be generated from YAML files and has gained significant popularity with over 300K downloads per month. Astronomer is now taking over DAG Factory, reinforcing its commitment to the open source community and working to integrate it more tightly with its own product experience and other tools.