December 2024 Summaries

Robinhood Markets, Inc., a leading financial services company, relies on Apache Airflow to manage its complex trading and operational workflows at scale. The company uses Airflow for diverse use cases across trading, clearing, money movement, and backend services, including charging margin interest daily, processing stock options after expiration, and managing cost basis adjustments. Robinhood employs a multi-cluster Airflow architecture with 15 clusters supporting different business domains, ensuring high availability and reliability through deployment of clusters across multiple Kubernetes environments and leveraging multi-AZ setups. The company has successfully migrated from Airflow 1.10 to Airflow 2, involving its 15 clusters, 75 workers, and over 4,000 data pipelines, and plans to enhance Airflow's integration with its internal authentication system and implement in-place upgrades with zero downtime for future enhancements.

Ford Motor Company has been a pioneer in the automotive industry, and recently, it increased its investments in AI and ML to improve customer experience, power its smart mobility strategy, and foster advances across the broader automotive industry. The company's ML engineers have leveraged Apache Airflow and Astro, a managed service from Astronomer, to process large amounts of data weekly, run parallel workflows, and reduce errors while accelerating AI model development for autonomous driving systems. Ford's journey with AI and ML was shaped by significant organizational and technical hurdles, but the establishment of its Artificial Intelligence Advancement Center and the development of specialized platforms like FordLLM have helped address these issues. The company has now transitioned to Airflow for its Mach1ML platform, which provides a robust orchestration tool that can handle hybrid setups and scale effectively, enabling faster time-to-value for ML solutions and reduced operational overhead.

Instacart, a leading grocery technology company, leverages Apache Airflow to orchestrate over 2,200 unique pipelines and 16 million tasks per month across its vast network of large and intricate data pipelines securely, compliantly, and at scale. The company's Data Infrastructure team has built a custom Airflow ecosystem with self-service authoring, infrastructure as code, monitoring, abstractions, and more to support its growth and scalability needs. With the help of Airflow, Instacart aims to maintain a healthy pace in migrating legacy pipelines to a central cluster, ensuring a smooth transition and optimal performance for its data operations.

The Airflow Summit 2024 featured two sessions from MLB teams showcasing how data pipelines orchestrated by Apache Airflow® enable fast decision-making and drive competitive advantage for players and coaches. The Texas Rangers use Airflow running on the Astro managed service to facilitate seamless data ingestion, complex ETL pipelines, and high quality, trustworthy data production. The Philadelphia Phillies leverage Machine Learning (ML) for player evaluation, acquisition, and development, using Apache Airflow® as the orchestrator for their entire ML lifecycle. Both teams have experienced significant improvements in efficiency, reliability, and scalability through their use of Airflow.

Data observability has become a critical aspect of modern data management as companies increasingly rely on data pipelines for their products or decision-making processes. Similar to supply chain management, data observability involves understanding the health and state of data across fragmented ecosystems. It can be broken down into three basic functions: analytics, monitoring, and alerting. Analytics focuses on the performance, status, and relationships between dependencies in a data product, such as tasks and datasets. Monitoring concerns data quality checks, task and pipeline run duration tracking, run status checking, and service-level agreements (SLAs) for tracking data freshness and timeliness. Alerting extends beyond reactive alerting to encompass proactive alerting, using analytics and monitoring to trigger alerts when anomalies are detected. Data observability is essential for companies to protect their critical data supply chains and enable teams to root-cause analyze issues quickly. It requires insight into who owns what up and down the supply chain, historical and predictive analytics for asset health, and data lineage tracking. Astro Observe, a new observability product, offers unified analytics, monitoring, and alerting features to help organizations manage their data pipelines effectively.

Panasonic Energy has been accelerating the transition to sustainable energy through the production of lithium-ion batteries since 2017. The company's data engineering team uses Apache Airflow to automate and structure its workflows, enabling the creation of live dashboards for operational analytics. Panasonic has gone through three iterations with Airflow, improving standardization, breaking down monolithic DAGs into manageable task groups, and implementing GitLab CI/CD pipelines for consistent deployment. The company is now exploring the future of Airflow with Kubernetes and streaming workflows to facilitate real-time data processing and seamless data streaming from Manufacturing Execution System (MES) and MQTT brokers.

This tutorial demonstrates how to create an ELT (Extract-Load-Transform) pipeline using AWS S3 and Databricks, two popular tools in data engineering. The process involves extracting data from S3, loading it into Databricks, and running transformations defined in Databricks Notebooks. This pattern is versatile and applicable across various industries such as FinTech, E-commerce, and B2C services. The tutorial provides step-by-step instructions on setting up the necessary connections between S3, Databricks, and Airflow, creating Databricks notebooks for data transformations, and deploying the ELT DAG to an Astro deployment.