Airflow in Action: Qualcomm Chip Design Orchestration at Million-Task Scale with Airflow 3

At the Airflow Summit, Qualcomm's Snapdragon CPU team detailed their transition from a Jenkins-based setup to using Apache Airflow for orchestrating chip design workflows at a high-performance computing (HPC) scale. The team utilizes Airflow to manage Electronic Design Automation (EDA) workloads, ensuring efficient resource allocation and task execution across globally distributed data centers. Their previous reliance on Jenkins resulted in challenges related to infrastructure maintenance and workflow consistency, prompting the switch to Airflow, which offers a stable, scalable platform with a user-friendly web interface. They implemented dynamic Celery workers, integrated with HPC schedulers like SLURM and LSF, to handle the extensive computational demands, and contributed improvements to the Celery CLI. By adopting Airflow's EdgeExecutor, Qualcomm managed to overcome the limitations of single data centers, enabling efficient workload distribution across multiple sites. The integration of Airflow into Qualcomm's workflows allows for comprehensive design verification, power estimation, and physical design processes, highlighting Airflow's potential as a unified orchestration layer for semiconductor workflows. Additionally, Astro's managed service offers Remote Execution, providing a secure, scalable solution that separates orchestration from execution, catering to organizations with strict compliance requirements.