July 2026 Summaries

Astro has introduced cross-region disaster recovery (DR) for Google Cloud Platform (GCP) data planes, enabling platform teams to pair primary and secondary clusters to ensure business-critical data pipelines remain operational even during regional outages. This DR capability, previously available on AWS, allows Airflow workloads to failover seamlessly with minimal downtime, adhering to a recovery time objective (RTO) of under one hour and a recovery point objective (RPO) of under 15 minutes. Unique to GCP, DR requires specific region pairings based on Google's supported dual-region configurations, impacting decisions on primary region selection. The infrastructure spans multiple availability zones and uses GCP's Cloud Storage dual-region buckets and Cloud SQL for PostgreSQL for state replication across regions. The implementation minimizes disruptions by maintaining consistent cluster IDs and configurations across failovers, with the DR system modeled as an extension of the existing cluster record. Networking on GCP demands planning for separate network ranges for the secondary cluster, and setting up DR involves a guided process with support from Astronomer for existing clusters. The capability is designed to be reliable and is continuously tested to ensure effectiveness during actual failover events, with Azure support planned for future expansion.

Astro has introduced a redesigned navigation system that provides a unified view of an entire organization's workspaces, addressing the previous friction caused by managing multiple workspaces separately. This new interface, currently available in Preview, displays deployments, Dags, IDE projects, and health data across all workspaces from a single view, making it easier for users to manage their Astro footprint without constant context-switching. The homepage now highlights essential information like recently failed Dags and deployment health, allowing users to quickly identify and address issues with features like failure rate sorting, direct investigation actions, and status filters. Users can also favorite important deployments and Dags, facilitating quick access to critical information without needing to open Airflow. The update aims to streamline workflow efficiency by providing a comprehensive organizational perspective and will become the default experience later this year, allowing users to opt-in now and provide feedback.

Airflow 3.3, released on July 6th, introduces significant features like the task state store and pluggable retries, aimed at enhancing the resilience and failure management of data pipelines. The task state store enables tasks to persist small pieces of information across retries, allowing for reconnection to ongoing external jobs without starting over, while pluggable retries allow users to customize retry policies based on the type of failure encountered. These updates are particularly beneficial given the emphasis on managing failures in data engineering. The release also includes improvements such as updates to asset partitions, a new language task SDK for Java and Go, and various UI enhancements, making it a compelling upgrade for users. With contributions from the Apache Airflow community and the first managed service support from Astronomer, Airflow 3.3 promises to streamline pipeline operations and reduce the time spent on failure management.

Apache Airflow is widely regarded as the standard for data orchestration due to its open-source nature, scalability, and extensibility, but the decision between self-managing it or using a cloud-managed service presents significant trade-offs for engineering teams. Self-managing Airflow can lead to high hidden costs in engineering time as teams deal with infrastructure maintenance, complex scaling issues, and the risk of losing critical expertise when key engineers leave. On the other hand, cloud-managed services, while reducing infrastructure costs, can introduce issues like vendor lock-in, always-on charges, and limited support for Airflow-specific problems. Astro, developed by Astronomer, provides a solution by offering a managed service that allows teams to focus on building pipelines rather than infrastructure maintenance, with benefits such as zero-downtime upgrades, built-in security compliance, and a hybrid architecture that maintains task execution within an organization's infrastructure. Organizations like Autodesk and WeWork have successfully transitioned to Astro, experiencing significant reductions in infrastructure management and troubleshooting time. Ultimately, the decision to self-manage or use a managed service like Astro should consider the actual engineering capacity and costs involved in maintaining Airflow.