October 2024 Summaries

Uber's data platform is essential to its business model, serving millions of customers and drivers across the globe. In 2016, Uber faced challenges managing multiple data workflow tools used by various teams, leading to inefficiencies and security risks. To address these issues, Uber standardized on Apache Airflow as a single workflow system due to its scalability, unified platform capabilities, and ease of use across teams. The company's internal Airflow fork, Piper, is now used by 1,000 teams with 200,000 different pipelines, orchestrating millions of tasks daily. Uber plans to contribute code back to the Apache Airflow project in its upcoming release.

The collaboration between Anyscale and Astronomer aims to provide a comprehensive solution for orchestrating and scaling machine learning and AI workflows. Anyscale offers a fully managed platform for deploying and scaling Ray clusters, while Astronomer is a modern data orchestration platform that enables organizations to streamline complex processes and integrate diverse tools. By combining Airflow's powerful workflow management with Ray's distributed computing capabilities, the partnership allows users to efficiently scale their machine learning workflows within the familiar Airflow environment.

The Apache Airflow Champions Program announces its final cohort for 2024, featuring data leaders from Fortune 500 companies worldwide. For the 2025 program, a new application process will be implemented to showcase exceptional candidates and limit onboarding to once a year. Cohort 4 of the Astronomer Champions Program includes nine Airflow enthusiasts from renowned organizations such as JPMorgan Chase, Gartner, and the Bank of Thailand. The cohort achieved a 30% acceptance rate, highlighting its diverse and experienced group ready to promote Apache Airflow globally.

Astronomer is launching a new Vulnerability Disclosure Program, partnering with Bugcrowd to better engage with the global security research community and identify vulnerabilities in their systems. The program aims to improve data security for customers by leveraging transparency, collaboration, and community involvement. Detailed guidelines and rules for participation can be found on Astronomer's Vulnerability Disclosure Program page: https://astronomer.io/vulnerability-disclosure.