Airflow in Action: Making dbt on Airflow Easy with Astronomer Cosmos. Insights from BAM

Lewis Macdonald and Ethan Stone from Balyasny Asset Management (BAM) presented at the Airflow Summit on how they optimized data transformations using dbt on Apache Airflow with Astronomer Cosmos. BAM's architecture, which manages data from thousands of sources, initially faced challenges related to complexity, team collaboration, and slow development velocity. To address these issues, BAM implemented a self-service dbt platform that uses Astronomer Cosmos to run dbt Core projects as Airflow DAGs and Task Groups, simplifying project bootstrapping, deployment, and execution with Kubernetes. This integration allows users to focus on SQL transformations without needing extensive Airflow experience, while providing enhanced observability and standardization across teams. The session highlighted the benefits of Cosmos, such as ease of use and improved production reliability, and looked forward to future enhancements with Airflow 3 and further optimizations for large-scale execution.