March 2023 Summaries

This blog post explores how to use Ray AIR to scale and accelerate the fine-tuning process of a stable diffusion model, a type of generative AI model that can convert textual descriptions into realistic images. The authors highlight three challenges when scaling fine-tuning diffusion models: converting scripts to do distributed training, distributed data loading, and distributed orchestration. To address these challenges, they introduce Ray AIR, a native set of scalable machine libraries built on top of Ray, which simplifies distributed training for PyTorch and other common ML frameworks, and provides an interface for reading files from cloud storage and efficiently loading and sharding data into training GPUs. The authors demonstrate how to use Ray AIR to fine-tune a stable diffusion model with ease, scalability, and minimal code changes, making it possible to put a cat on the moon!

The text discusses how two open-source frameworks, Alpa and Ray, integrate to achieve scale in training large language models (LLMs) like OPT-175B with pipeline parallelism up to 1024 A100 GPUs. Alpa automatically discovers and executes the best inter-op and intra-op parallelism for LLMs, while Ray is a unified framework for scaling AI and Python applications like machine learning. The integration of Alpa and Ray enables efficient training and inference of LLMs at scale, reducing scheduling frequency and overhead, and achieving high performance and scalability results, including peak HW FLOPs utilization of ~57.5% and ~179 TFLOPs/GPU.

Ray is being used by leading AI organizations to train large language models at scale, including OpenAI, Cohere, EleutherAI, and Alpa, to support the production deployments of generative model workloads. Generative image and language models require significant computational resources, making distributed training and deployment essential for supporting these workloads in production. Ray provides a flexible solution for scaling ML workloads, tackling challenges such as partitioning models across multiple accelerators, setting up training to be tolerant of failures on preemptible instances, and deploying models that span multiple GPUs on multiple nodes. The framework also supports scale-out strategies, where users can run many copies of a workload to deploy an online inference, fine-tuning, or training service at a lower cost than running a single high-end device. Ray Core scheduling enables the orchestration of large-scale distributed computations required for training generative models from scratch, while Ray Train and Ray Serve provide out-of-the-box Trainer classes and APIs for scaling model deployment graphs, respectively. The framework is being enhanced with new features, including streaming batch inference support, async requests in Ray Serve, and integrations with popular frameworks such as HuggingFace Accelerate, DeepSpeed, and Alpa to Train.

Ray Summit stories showcase how users are using Ray to scale various applications, including training large language models, building ML platforms, and scaling parallel Python jobs. Instacart migrated from an AWS ECS stack to Ray, achieving a 10x better cost performance and simplifying deployment. Dow used Ray's multi-agent decomposition approach to solve a complex production schedule design problem, reducing computation time by 10x. Ridecell leverages Ray Tune to efficiently search across hyperparameters for their deep learning models, optimizing trials and preventing costly mistakes. These examples demonstrate the power of Ray in scaling applications and solving complex problems.

Uber's Michelangelo platform, built on top of Ray, achieved a 50% savings in ML compute costs for large-scale deep learning jobs by using a heterogeneous (CPU + GPU) cluster. The Uber team also experienced a 4x speedup in hyperparameter tuning jobs using Ray Tune. Spotify's ML team chose Ray due to its rich ML ecosystem integration and simplicity, eliminating the need to learn other frameworks or APIs. The team was able to democratize their platform, making it more accessible to employees from various backgrounds. Shopify's ML platform team built on top of open-source projects like Kubernetes and Ray, prioritizing scalability, fast iterations, and flexibility. By focusing on real use cases and user experience, the teams were able to successfully scale their ML workloads and deliver innovations with Ray.

The OOM monitor is an out-of-memory (OOM) detection and prevention feature in Ray, designed to prevent memory-intensive tasks and actors from consuming excessive resources and causing cluster-wide resource degradation. This feature aims to provide better Python support for detecting memory usage mechanisms while using Ray native libraries or third-party Python libraries with Ray, enabling machine learning engineers to observe and debug their applications more effectively. The monitor works by periodically inspecting collective memory usage on each worker node, terminating a task or actor as a preventive measure before an OOM event occurs, and rescheduling it later if necessary. The policy applied to decide which task to free up for memory is multi-step, prioritizing retriable tasks, grouping tasks by caller, picking one task from the group, and ensuring fairness among callers. The feature is enabled by default in Ray 2.2 and 2.3, providing a novel solution to prevent OOM errors and improve observability and transparency of ML workloads.