February 2025 Summaries

The announcement of the first beta build of Apache Airflow 3.0 marks a significant milestone for the project, showcasing extensive community collaboration and development over four years. This release introduces major updates, including DAG versioning, enhanced backfill support, and the Task Execution Interface, which aims to enable execution in any environment and language, facilitating multi-cloud deployments. The UI has been modernized with React and FastAPI, and significant security enhancements have been implemented. The beta is intended for testing by developers, with general availability expected by mid-April 2025, and community feedback is highly encouraged to refine the final release.

Apache Airflow has firmly established itself as the industry standard for data orchestration, with a surge in popularity and widespread adoption. The platform boasts over 31 million monthly downloads, 3,000+ contributors, 29,000 pull requests, and 77,000 organizations using Airflow. The State of Airflow Report 2025 captures insights from over 5,000 data professionals, revealing that Airflow is becoming the foundation for success in modern data ecosystems. It's indispensable, with over 90% of surveyed engineers recommending it as a foundational tool. Enterprise adoption is scaling rapidly, with large enterprises running over 20 production instances and handling higher data volumes. Airflow is driving direct business impact, with over 85% of users expecting an increase in external-facing or revenue-generating solutions built on the platform. The report also highlights growing reliance on Airflow for AI and machine learning workloads, as well as its flexibility in a multi-cloud world. With the upcoming release of Airflow 3.0, the future of data orchestration is brighter than ever, with features designed to make it more secure, intuitive, and responsive to data needs.

Meteosim, a company specializing in meteorological and environmental services, has successfully integrated Apache Airflow with the Slurm workload manager to streamline their high-performance computing (HPC) workflows. This integration enables the company to orchestrate complex simulations across multiple Slurm-managed compute clusters while optimizing resource utilization, maintaining service uptime, and simplifying data pipeline creation. Prior to adopting Airflow, Meteosim faced challenges with Crontab chaos and manual monitoring tools, which drove them to adopt a scalable solution for orchestrating their pipelines. The integration architecture features deferrable operators, custom-built integrations, and Redis as a messaging layer to manage job states and ensure reliability. This innovative use of Airflow and Slurm has delivered significant benefits, including scalability, reliability, ease of use, and continuous improvement, resulting in zero downtime across 6,000 pipelines.

Airflow is being used by Laurel, a Series B startup, to orchestrate and optimize GenAI models for automating timesheet creation in their legal and accounting firm. The company leverages Airflow's modular DAGs to automate daily model retraining, deployment, and inference scheduling, which has resulted in significant savings of over $500k per year. Airflow helps Laurel standardize its ML pipeline, delivering safe rollouts, rapid iteration, and improved system performance. Additionally, Astro, a fully managed Airflow service, has enabled the company to automate manual workloads, reduce costs, and scale efficiently, making Airflow and Astro the backbone of their GenAI workflows.

Astro Observe is now generally available, offering a unified approach to orchestration and observability that helps data teams ensure data products meet SLAs, catch failures before they impact the business, diagnose issues in minutes, optimize pipeline performance and cost-efficiency, and eliminate reactive troubleshooting. With its Data Health Dashboard, Astro Observe provides real-time visibility into pipeline health, predictive insights to prevent failures, and automated diagnostics to resolve issues faster. By combining deep observability with Airflow orchestration, Astro Observe gives teams a single pane of glass into data product health and performance for modern data teams, enabling them to proactively manage their pipelines and move beyond reactive troubleshooting into proactive, strategic data operations.

FreightWaves, a company that provides real-time market intelligence for the logistics industry, faced significant challenges in managing their data engineering operations. They struggled with a scattered microservices setup, painful development cycles, and operational risks due to lack of workflow reliability. To address these issues, they migrated to Astronomer's Astro, a fully managed Apache Airflow service, which provided a rich ecosystem of provider packages for integrating with diverse data sources, local development environments for faster iterations, and centralized workflow management to simplify scaling and maintenance. The migration enabled FreightWaves to streamline their integration of new data partners, enhance developer productivity, and eliminate downtime in daily updates, ensuring customers received uninterrupted insights. With Astronomer's support, FreightWaves was able to transform their data engineering practices, unlock resources for innovation and experimentation, and onboard new data sources seamlessly.

Rakuten Kobo, a Canadian company acquired by Rakuten in 2012, has effectively leveraged Apache Airflow to support its diverse teams, including data science, finance, marketing, and customer support, by maintaining a single shared environment. At the 2024 Airflow Summit, Spencer Tollefson, Team Lead Data Engineering at Rakuten Kobo, detailed their strategy for enabling team self-service, which includes delineating responsibilities, establishing guardrails for new developers, and implementing scalable monitoring and access control systems. The company's approach has minimized bottlenecks and improved collaboration by clearly defining roles, such as data engineers maintaining the environment and business developers focusing on DAG creation with support. Kobo's system includes comprehensive documentation, local development environments, and staging for realistic testing, while their alert system ensures rapid response to failures. Looking forward, Kobo plans to explore YAML-based DAG authoring and multitenancy features in Airflow to further enhance scalability and separation between teams, aligning with upcoming advancements in Airflow 3.0 for improved security and task isolation.