Airflow in Action: Best Practices Learned From Scaling AI at Oracle

At the Airflow Summit, Ashok Prakash, Senior Principal Engineer at Oracle, discussed the challenges and solutions in building and operating high-scale AI systems, particularly focusing on the role of Apache Airflow in managing these systems on cloud platforms like Kubernetes. He emphasized the importance of separating infrastructure provisioning from orchestration to prevent bottlenecks, using tools like Terraform for platform provisioning, and employing Airflow as a logic control plane to manage GPU-driven workloads effectively. Prakash highlighted that MLOps involves heterogeneous workflows, and Airflow's ability to handle diverse data types, coordinate scalable compute resources, and ensure operational reliability makes it crucial for AI and ML pipelines. He also provided insights into optimizing GPU usage through dynamic workflows, ensuring operational efficiency and maximizing return on investment. The presentation concluded with a focus on production fundamentals, including CI/CD for workflows, modular Dag design, and security practices, underscoring that successful AI scaling requires coordinated management of compute, data, and teams.