October 2025 Summaries

Ray, an AI compute platform, has introduced label selectors to enhance scheduling flexibility by allowing developers to specify where workloads should run based on node characteristics. Released in Ray 2.49, this feature is available across Ray Dashboard, KubeRay, and Anyscale, enabling users to assign labels to nodes, such as CPU type or market type (spot or on-demand), and use these labels to direct task and resource placement more efficiently. The new API addresses prior limitations by providing a more intuitive way to express scheduling requirements without resorting to hacks, improving developer experience and debugging. Inspired by Kubernetes labels and selectors, the integration fosters interoperability between Ray and Kubernetes, potentially unlocking advanced use cases. The system supports both static and autoscaling clusters, with future plans to extend label selector functionalities to Ray libraries and improve Kubernetes interoperability further.

Ray, an open-source distributed compute engine, is joining the PyTorch Foundation under the Linux Foundation to further expand its global community and meet the growing need for scalable AI infrastructure. Initially developed at UC Berkeley to address the challenges of scaling AI workloads across CPU and GPU clusters, Ray's adoption has dramatically increased, with the tool now powering AI workloads for major companies like Netflix, Apple, and JPMorgan. The surge in demand is largely attributed to advancements in generative AI and large language models, prompting Ray to become a pivotal component in the AI compute stack alongside other foundational open-source projects like PyTorch and Kubernetes. As Ray continues to build its community and enhance its offerings, its integration into the PyTorch Foundation marks a significant step in its journey towards becoming a standard in distributed AI computing.

Ray, a distributed compute framework that unifies the AI ecosystem, is increasingly being adopted by industry leaders like Attentive and Shopify for building AI at scale. In response to the growing demand for professionals skilled in Ray, a new Ray Foundations Certification has been launched to validate expertise in its core architecture, primitives, and libraries. This certification aims to enhance career growth for ML Engineers, Data Scientists, Data Engineers, Platform/Infra Engineers, and AI Researchers by proving their ability to build scalable, production-ready AI systems. It requires no prerequisites and offers self-paced courses for preparation. The online exam consists of 60 multiple-choice questions and covers various aspects of Ray, including Ray clusters, architecture, and different Ray libraries.

Multimodal AI workloads, which involve processing diverse data types like text, images, audio, and video, are increasingly challenging current infrastructure capacities, requiring systems capable of managing high-throughput pipelines and efficient CPU and GPU scheduling. Ray Data, a data processing engine designed for these workloads, offers a streaming batch execution model that optimizes resource utilization and reduces costs, alongside ecosystem integrations with AI projects like vLLM and PyArrow. It scales CPU and GPU workers independently and supports fault tolerance and autoscaling, allowing code to run unchanged across various scales. Recent benchmarks compared Ray Data to Daft, a distributed DataFrame library, revealing Ray Data to be generally faster and more efficient, particularly in leveraging GPU utilization and reducing CPU starvation. The benchmarks highlighted Ray Data's superior performance, especially in large-scale workloads, where it achieved up to 7x faster processing than alternatives. Ray Data's design for cluster heterogeneity enables it to maximize GPU efficiency in pipelines with substantial CPU steps, and its ongoing optimizations promise further advancements. While Daft is acknowledged for its efficiency in low-resource settings, Ray Data demonstrates enhanced performance with larger instance types, encouraging users to evaluate both systems based on specific needs.

Robert Nishihara and Philipp Moritz discuss the use of Tinker, a new API from Thinking Machines, designed for training large language models (LLMs), and its application with Ray to develop a text-to-SQL model. The process involves two main stages: data generation and model fine-tuning. Data generation is achieved by using Ray Serve and vLLM to deploy Qwen-8B for generating SQL queries at scale, with Ray Core executing parallel tasks to evaluate these queries in a SQL environment, filtering out unsuccessful ones. The successful queries are then used in the model fine-tuning phase with Tinker, which provides granular control over the training process, allowing for the fine-tuning of LLMs using a dataset created in the first stage. The authors also describe the evaluation of the model’s performance, detailing the extraction and merging of LoRA weights with the base model to address compatibility issues with vLLM. The article provides a comprehensive look at leveraging cutting-edge tools for optimizing AI model deployment and fine-tuning, supported by a detailed code appendix for implementation.