January 2025 Summaries

The text discusses the evolution of the modern data stack, emphasizing the need for orchestration and DataOps to address challenges in scalability and interoperability that traditional systems have struggled with in the wake of increasing data volumes and complexity. Companies like Snowflake and Databricks have thrived by offering scalable data compute platforms that integrate seamlessly with cloud services. However, the data stack remains fragmented, particularly in the DataOps layer, which is crucial for turning raw data into valuable business products. Orchestration is highlighted as a key component for unifying data operations, offering control, management, and integration across various tools and data sources. Apache Airflow is cited as the leading orchestration tool, widely adopted by major companies for its scalability and flexibility. Astronomer, leveraging Airflow, aims to lead the DataOps space by integrating observability and AI-driven enhancements into its platform, Astro, to streamline data operations and enhance business value.

The Astronomer Data Excellence Awards were created to celebrate the innovative achievements of teams using Astro and Apache Airflow in data orchestration. The awards recognize groundbreaking efforts in areas such as AI initiatives and large-scale data operations, with winners including Marriott Hotel and Resorts, Booking.com, Vibrant Planet, Wynn Resorts, and Foursquare. These organizations have demonstrated the power of combining effective tools with creative thinking to tackle complex data engineering challenges and drive real business value. Astronomer is committed to enhancing its platform, Astro, to support over 700 enterprises in achieving data excellence, inspired by the accomplishments of these award-winning teams.

Lewis Macdonald and Ethan Stone from Balyasny Asset Management (BAM) presented at the Airflow Summit on how they optimized data transformations using dbt on Apache Airflow with Astronomer Cosmos. BAM's architecture, which manages data from thousands of sources, initially faced challenges related to complexity, team collaboration, and slow development velocity. To address these issues, BAM implemented a self-service dbt platform that uses Astronomer Cosmos to run dbt Core projects as Airflow DAGs and Task Groups, simplifying project bootstrapping, deployment, and execution with Kubernetes. This integration allows users to focus on SQL transformations without needing extensive Airflow experience, while providing enhanced observability and standardization across teams. The session highlighted the benefits of Cosmos, such as ease of use and improved production reliability, and looked forward to future enhancements with Airflow 3 and further optimizations for large-scale execution.

Migrating legacy ETL workloads from Informatica to Airflow using DAG Factory offers significant benefits, including flexibility, scalability, and robust monitoring. By leveraging YAML configurations to define DAGs, organizations can streamline the process of converting legacy workflows without needing extensive custom code. This approach enables data teams to efficiently adopt Airflow's modern orchestration capabilities, simplifying the journey to a more adaptable and observable ETL environment. With tools like DAG Factory, organizations can dynamically create Airflow DAGs from configuration files, automating up to 90% of the process, and ultimately position themselves to handle future data workloads with confidence and efficiency.

At the Airflow Summit 2024, Bhavesh Jaisinghani, Data Engineering Manager at Autodesk, shared how his team transformed testing of their data pipelines by creating a secure production-like UAT environment with Astronomer and Apache Airflow. This setup enabled seamless testing of Spark-based workflows, reduced development cycles, and ensured data security for sensitive PII. Autodesk's data platform consists of four primary layers: ingestion, transformation, data warehousing, and BI / reporting, relying heavily on Apache Spark for data processing. The company implemented a UAT environment mirroring production, featuring separate infrastructure, granular access control, data sync utilities, and CI/CD integration. This setup delivered impressive results, including improved data quality by 90%, reduced development cycles by 33%, and technical debt reduction. Autodesk's innovative approach powered by Astronomer and Airflow showcases how secure, scalable UAT environments enable faster, more reliable pipeline development.

TrackFly, a Utah-based startup providing marketing insights and supply chain visibility to small retail shops in the outdoor goods industry, struggled with managing Airflow infrastructure independently due to resource constraints and business risk. They adopted Astronomer's fully managed service for Apache Airflow, which eliminated operational overhead, streamlined development workflows, and enabled scalable automation of machine learning pipelines. As a result, TrackFly achieved faster deployments, consistent uptime, and actionable insights, ultimately improving customer satisfaction and scaling its offerings to meet growing industry demands.

Wix leverages Apache Airflow as the backend for its machine learning platform, extending its use beyond traditional pipeline scheduling to streamline ML workflows and automate operational tasks. Wix's data science practice runs over 200 machine learning models daily for tasks such as fraud detection, text embeddings, and classification, with an estimated 10% of the world's websites powered by Wix. The company uses Airflow's API and Python libraries to integrate it seamlessly into its ML platform, supporting on-demand DAGs, programmatic cancellation, real-time monitoring, and scalability. Wix's implementation demonstrates how Airflow can be used beyond traditional scheduling to automate machine learning workflows, offering a proven blueprint for integrating Airflow into complex machine learning systems. The upcoming Apache Airflow 3.0 release further expands the flexibility engineering teams have in running their data pipelines, and Astro, the industry-leading Airflow managed service, offers a free trial option for getting started with Airflow for MLOps or other workflow orchestration use cases.

Cloudflare, a global cloud services provider, is leveraging Apache Airflow to orchestrate the provisioning, diagnostics, and recovery of its massive infrastructure across 330 cities in 120+ countries. To address operational challenges, Cloudflare developed Phoenix, an autonomous system that uses Airflow to discover, diagnose, and recover servers worldwide, automating workflows from powering on broken servers to running diagnostics with custom Linux images. Additionally, the company has extended Airflow's capabilities with Zero Touch Provisioning (ZTP), enabling rapid deployment of inference-optimized GPUs across its network, significantly reducing deployment times and allowing for rapid scaling of AI and machine learning infrastructure. Through its innovative use of Airflow, Cloudflare is optimizing its global infrastructure to meet growing demand for AI and machine learning capabilities.

Apache Airflow was successfully used by Circle, a digital financial technology company, to manage their blockchain data orchestration and scale their data platform. Circle initially implemented CI/CD practices, automated testing frameworks, and enhanced data quality checks to overcome challenges in their managed Airflow environment. Despite these improvements, they realized the limits of performance with Managed Workflows for Apache Airflow (MWAA) and began exploring alternative managed Airflow solutions like Astro from Astronomer.io. Circle is now evaluating Astro due to its support for the Kubernetes Executor, which offers flexibility in using containerized Airflow environments, as well as features such as seamless local development, streamlined environments, and data observability with Astro Observe.