November 2024 Summaries

ASAPP, an AI leader in the contact center industry, has successfully reduced workflow runtimes by 85% using Apache Airflow® and custom Apache Spark® solutions. The company's Data Ops and MLOps ecosystem is built to support continuous innovation, demanding deep analytics along with frequent retraining and fine-tuning of ML models. ASAPP leverages Airflow for data engineering, DataOps, and ML & LLM model training and evaluation. By integrating Spark for massive scaling, the company has significantly improved its ASR workflow runtimes from 43 hours to under 5 hours. Key learnings from ASAPP's experience with Airflow include strong orchestration foundation, integrated Spark support, and adaptability for LLM workflows.

LinkedIn uses Apache Airflow for orchestrating its systems deployment pipelines, handling one million deployments per month across 7,000 services. The company's Continuous Deployment (LCD) infrastructure platform aims to improve the deployment experience and reduce developer effort. LCD enables developers to declare their pipelines and dependencies through a low code/no code UI, which translates into Airflow DAGs that orchestrate the deployment pipeline and automate validation steps. With upcoming support for building and running tasks in any language, Airflow 3.0 will further break down barriers to developer adoption.

Stripe, the payment infrastructure of the internet, processes $1 trillion in payments annually. To ensure regulatory compliance and maintain data integrity while enabling developers to innovate quickly, Stripe developed User Scope Mode (USM), an internal tool that allows safe testing of Apache Airflow® data pipelines without risking production data corruption. Stripe operates Airflow at a massive scale, processing multiple petabytes of data daily and managing 250 complex pipelines with 150,000 tasks. The company is transitioning from its own Airflow fork to the mainline project for reduced engineering efforts and faster access to new features. USM has revolutionized Stripe's development and testing workflows by enabling efficient validation of pipelines while maintaining strict compliance requirements, permissioning, and data integrity.

The Bosch Group, a leading global technology supplier, uses Apache Airflow® to handle 1.2 million pipeline runs per day in its data-driven applications and analytics. Initially designed to manage 1,000 DAG runs per hour, the company faced challenges due to frequent event storms that impacted data freshness. To address this issue, Bosch scaled Airflow by iterating through various improvements such as tuning hardware, upgrading config parameters, and optimizing DAGs and environments. As a result, they achieved 50,000 DAG runs per hour with an average of 1.7 million events processed daily and single-second latency.

Autodesk, a global leader in design software, has successfully transformed its data engineering processes by leveraging Astronomer and Apache Airflow. The company's previous orchestration tool, Apache Oozie, presented challenges such as complex user interface, high maintenance overhead, scalability issues, and development bottlenecks. By migrating to Astro, Autodesk benefited from a more intuitive user interface, reduced maintenance time, seamless scaling of workloads, and the ability to create independent development environments. The transition has led to increased operational efficiency, enhanced scalability, minimal workflow disruptions, and improved business impact through timely data delivery and better visibility into team performance.

The text discusses the importance of being selective when setting up an observability solution using Airflow and Astro. It highlights four key insights to consider focusing on: data freshness, on-time delivery, data dependencies tracking, and data quality. The author introduces Astro Observe, a new product that enables users to leverage these observability insights with control over the assets they monitor, the insights they receive, and who has access to what. Astro Observe uses Data Products to allow users to create customized, focused views of metadata emitted by Airflow. The text also provides an overview of how Astro Observe can be used for monitoring data freshness, on-time delivery, data dependencies tracking, and data quality.

Neha Singla and Sathish Kumar Thangaraj, senior software engineers on Apple’s Data Platform team, presented a session at this year's Airflow Summit demonstrating how they used Jupyter notebooks and Apache Airflow to streamline data science workflows. They tackled common bottlenecks in transitioning experiments from prototype to production, resulting in increased productivity, simplified debugging, and support for large-scale, distributed workflows. The solution involved using the Papermill operator to parameterize and execute Jupyter Notebooks, which was extended by Apple engineers to support multiple languages and runtimes, as well as remote kernels running in Kubernetes clusters. This approach has delivered tangible benefits such as enhanced productivity, scalability, and improved debugging. Looking ahead, the team aims to further enhance their solution by supporting event-driven notebook workflows, improving workflow sharing, and collaborating with the open-source community to expand capabilities.

Burns & McDonnell, a global design firm, faced challenges with its existing data platform due to disconnected systems and lack of metadata. The company used Apache Airflow® as an orchestrator to create a scalable and trustworthy data platform. This transformation allowed the data engineering team to turn around requests from the business in less than 24 hours, providing reliable, trustworthy, and accessible data across the company.

Bloomberg, a leading financial data provider, transitioned from manual processes to an automated data orchestration framework using Apache Airflow to handle its mortgage ETL pipeline. The previous workflow was labor-intensive, complex, and inefficient, with high risks of failure and limited observability. By adopting Apache Airflow, Bloomberg achieved a 51% reduction in run time, eliminated key person risk, and improved workflow monitoring. The event-driven ETL pipeline now comprises over 100 tasks, using various Airflow operators and custom implementations.

Proactive Airflow Monitoring is crucial to prevent infrastructure issues before they affect production pipelines. Deployment Health Alerts, offered by Astro, provide out-of-the-box, automated alerts that give teams immediate visibility into their Airflow infrastructure's health. These alerts monitor key deployment components and send actionable notifications when issues arise, helping teams avoid business-critical failures while maintaining operational efficiency. Deployment Health Alerting is automatically turned on for new deployments in Astro, ensuring reliable pipelines without additional configuration.