August 2026 Summaries

Apache Airflow has evolved from a batch-workflow scheduler into a platform for coordinating large-scale data and AI workloads, but its traditional polling-based scheduling and Celery-based execution path can add delays as task volumes grow. Astronomer describes how its managed Astro platform preserves Airflow’s Dags, task states, operators, metadata database, and Airflow 3 Task SDK interfaces while replacing selected internal components with an event-driven Astro Scheduler, direct-assignment Astro Executor, and Airflow-aware Astro Hypervisor. Rather than relying primarily on periodic discovery loops, these services use small brokered “hints” to prompt database-backed scheduling and task assignment, while retaining database scans as a recovery fallback. Workers long-poll the API server for assignments, use pre-warmed processes to reduce startup cost, and report execution through Airflow’s API, while the Hypervisor uses workflow state alongside Kubernetes and service-health signals to scale components, repair known operational problems, and manage hibernation. Astro also includes maintained runtime images and optional cross-region recovery with a stated recovery point objective below 15 minutes and recovery time objective below one hour. In Astronomer’s load tests, the combined scheduler and executor reportedly sustained 500,000 concurrent tasks and achieved 228-millisecond p95 task-start latency at 100,000 concurrent tasks, compared with 23.582 seconds for a standard Airflow scheduler with Celery at 50,000 tasks, although the results depend on infrastructure, workload characteristics, and configuration.

Astronomer announces major performance and reliability enhancements to its managed Astro platform, re-engineering key Apache Airflow scheduling, execution, scaling, and recovery components while retaining Airflow compatibility without requiring customer changes. The platform is designed for modern data and AI orchestration workloads, including ETL, dbt transformations, model training and evaluation, and event-driven pipelines, and is reported to support up to 500,000 concurrent tasks. In load tests, Astro achieved p95 task-start latency of about 230 milliseconds at 100,000 concurrent tasks and roughly 300 milliseconds at 300,000 tasks, compared with Celery’s reported 23.582-second latency at 50,000 tasks. Astro attributes these results to event-driven scheduling, an Airflow-aware scaling and healing layer, high-availability components, and multi-region database recovery capabilities, while claiming 85% fewer failures under load than open-source Airflow.

Astronomer has made cross-region disaster recovery for Astro generally available on Azure, completing its rollout across AWS, Google Cloud, and Azure for Enterprise Business Critical customers. The feature allows customers to enable a secondary Azure region through the UI, API, or Terraform, with Astro managing replication of Airflow metadata through Azure PostgreSQL geo-replicas, task logs and container images through Blob Storage object replication, and a warm-standby AKS environment designed to restore workloads in under one hour with a target recovery point objective of 15 minutes for supported task-log region pairs. Astro retains a one-cluster, two-region control-plane model so cluster identities, hostnames, and customer-facing endpoints remain unchanged after failover, while DNS redirects traffic to the promoted region. Azure’s replication capabilities allow DR to be enabled on existing clusters without downtime or separate data migrations, though customers must prepare secondary-region workload identities, private networking connections, registry secrets, and any custom routing within their own Azure tenants. Replication health is monitored directly using Azure storage policy status and PostgreSQL replication-delay metrics, and failover promotes the database replica, creates workloads on the standby cluster, updates DNS, and establishes reverse replication for eventual failback.