Airflow in Action: ETL Insights from Bloomberg — Slashing Runtimes by 50%

Bloomberg's Data Platform Engineering team, responsible for managing over 2,500 dynamic Dags to deliver alternative financial data to Bloomberg Terminal subscribers, showcased their innovative use of Apache Airflow configurations at the Airflow Summit. Software engineers Yu Lung Law and Ivan Sayapin explained how they addressed four production challenges without writing new code by fine-tuning native Airflow parameters. These adjustments improved Dag parsing time, resolved a scheduler memory leak, and enhanced their monitoring through StatsD integration. The team emphasized the importance of testing configuration changes in lower environments, using temporary fixes while investigating long-term solutions, and staying updated with the latest Airflow capabilities. They also highlighted the benefits of using Astro, a managed Airflow platform by Astronomer, which simplifies monitoring and observability, allowing teams to focus on data product development with real-time insights and streamlined operations.