June 2026 Summaries

Ray Data 2.56 focuses on addressing key reliability issues in AI data pipelines, specifically out-of-memory (OOM) failures and unnecessary object spilling, which previously led to job crashes and slowdowns. By introducing memory-aware execution, improved memory registration, and enhanced process management, the update successfully reduces OOM errors, leading to zero such errors and significant runtime improvements in internal tests. Additionally, the update consolidates block formats to PyArrow and optimizes prefetching processes, mitigating object spilling and reducing peak object store memory usage, while also boosting training throughput. The version also introduces multiple dataset support, enhances training shuffle performance, and improves scheduling loop scalability, all aimed at providing a more reliable and efficient user experience. Looking ahead to version 2.57, further stability improvements are planned, including better handling of prefetched training batches and fault-tolerant shuffles.

The blog post discusses the challenges and solutions involved in evaluating robot foundation models using Ray and Anyscale. It highlights the necessity of policy evaluation in robotics, where real-robot testing can be slow, costly, and unsafe, making simulation a vital alternative due to its safety, repeatability, and cost-effectiveness. However, simulation introduces a distributed-systems problem, requiring the coordination of GPU-heavy simulation and policy inference across numerous rollouts. Ray and Anyscale address three main challenges: ensuring closed-loop execution within simulation, separating GPU workloads to prevent resource contention, and efficiently sharing policy replicas across multiple rollouts. By disaggregating simulators and policy inference into separate processes and scaling them independently with Ray Serve, the solution enables parallel evaluation at scale, transforming evaluation into a scalable service akin to modern AI workloads. This approach allows for seamless scaling across GPU clusters, leveraging Ray’s primitives for distributed inference and offering a reproducible, flexible infrastructure for evaluating robotics models.

Ray Serve LLM, in collaboration with the Google Kubernetes Engine (GKE) team, has achieved a significant enhancement in throughput and latency by implementing architectural upgrades such as direct streaming, a new vLLM Ray executor backend, and HAProxy integration. These optimizations have enabled up to 4.4x higher request throughput on prefill-heavy workloads and up to 24x on decode-heavy workloads. The enhancements allow Ray Serve LLM to match the performance of the rust-based vllm-router framework, showcasing improvements in orchestration overhead and efficiency. By leveraging Ray's primitives for fault tolerance and observability, the updates facilitate complex distributed computing pipelines across Kubernetes and VMs. The new direct streaming mode decouples the request routing control plane from the request/response streaming data plane, resolving bottlenecks in the legacy architecture and optimizing performance in multi-replica and agentic workloads. These developments position Ray Serve LLM as a robust solution for high-performance distributed inference, maintaining parity with existing frameworks while preserving Ray's flexibility and scalability capabilities.

Data processing is increasingly transitioning from traditional CPU-based systems to GPU-based systems as organizations seek to extract value from various forms of unstructured, multimodal data such as videos, audio, and sensor outputs. This shift is driven by the need for model inference to process and structure these complex data types, which are not suitable for SQL-based manipulation. As a result, there is a growing reliance on GPUs to handle inference-heavy workloads, enabling new insights and efficiencies from data that were previously difficult to utilize. This evolution introduces challenges in system architecture, as traditional homogeneous clusters can lead to underutilization and inefficiencies, prompting the development of solutions like Ray and Anyscale to optimize resource allocation and execution. The need for streaming execution, handling API-bound stages, and managing large-scale batch inference further highlights the complexities of modern data pipelines, with companies like Netflix, Alibaba, and Nvidia pioneering frameworks to support these advanced workflows.

In the exploration of Prefill-Decode (PD) disaggregation using Ray Serve LLM on AMD hardware, this blog post discusses how it can significantly enhance the performance of LLM serving by achieving up to 2.7x better "goodput," translating into cost savings of up to 67%. PD disaggregation separates prefill and decode phases onto dedicated GPUs, thereby eliminating mutual interference and enabling each phase to run closer to its theoretical throughput. While it offers advantages such as consistent TPOT under load and compounded savings over long output sequences, it also introduces operational complexities like KV cache transfer and workload-specific tuning of the prefill-to-decode ratio. The post highlights scenarios where PD is beneficial, particularly for TPOT- or E2E-sensitive workloads, and where aggregated serving is preferable, especially when TTFT is a critical constraint. The blog provides insights into the use of RIXL for KV transfer on AMD MI325X and emphasizes the importance of matching the P:D ratio to workload demands to avoid the potential pitfalls of PD disaggregation.

This blog post by Suman Debnath explores the intricacies of Fully Sharded Data Parallelism (FSDP) in distributed model training, utilizing PyTorch and Ray. It explains how FSDP efficiently manages memory and improves GPU utilization by sharding model parameters, gradients, and optimizer states across multiple GPUs, overcoming the limitations of traditional data parallelism. The article provides a step-by-step walkthrough of FSDP's operation, focusing on its two-dimensional splitting strategy—vertical partitioning of model layers and horizontal sharding across GPUs. It discusses the implementation of FSDP with PyTorch and Ray Train, highlighting the advancements in FSDP's second version, FSDP2. Additionally, the post introduces DeepSpeed as an alternative framework for distributed training and demonstrates a practical application of these concepts by fine-tuning a large text-to-speech model, Qwen3-TTS, to clone a unique voice. The article underscores the balance FSDP strikes between memory efficiency and communication overhead, enabling the training of larger models with enhanced scalability.

The blog post by Aydin Abiar and Philip Wang describes the process of debugging a Ray Data and vLLM pipeline using Anyscale's new agent skills, specifically focusing on a video-captioning issue. The authors faced a startup problem due to a KV cache memory error with a vision-language model, which they resolved using Anyscale's platform skills such as /anyscale-platform-fix, /anyscale-platform-inspect, and /anyscale-platform-run. These skills streamline debugging by automating static validation, runtime inspection, and workload deployment on Anyscale, allowing users to efficiently identify and fix issues with minimal human intervention. The authors demonstrate how Anyscale's platform skills reduce the manual effort required for debugging by automating the process, which includes inspecting logs, validating code, and suggesting solutions, ultimately saving significant time and effort. Through the walkthrough, they illustrate the ease and effectiveness of resolving complex issues using these tools, highlighting the reduction in time spent from hours to mere minutes of active human decision-making.

Ray Day London, the final stop of Anyscale's 2026 Ray on the Road series, highlighted the advancements in AI infrastructure facilitated by Ray, an open-source framework for distributed computing. The event showcased various industry applications demonstrating Ray's ability to efficiently scale and simplify complex AI workloads. Xoople utilized Ray to perform rapid geospatial predictions over vast areas, while Adyen employed it to train a Transaction Foundation Model on a massive 51 trillion token dataset for fraud detection. Criteo enhanced its ad relevancy model's scalability and speed by integrating multimodal capabilities, achieving a 50x inference speedup. BMW's AI Gateway expanded from text to video processing using Ray and vLLM, leveraging cost efficiencies through self-hosted models. The event underscored Ray's flexibility and impact in facilitating scalable, reliable AI solutions across diverse domains.

Ray Day: NYC, a key event in Anyscale's 2026 Ray on the Road series, brought together AI practitioners, Ray developers, and the Anyscale team to explore the intricacies of scaling AI workloads using Ray. The event featured presentations from companies like Torc Robotics, Discord, Cubist/Point72, and Coinbase, each sharing their experiences of transitioning to Ray to overcome challenges in AI infrastructure. Torc Robotics highlighted their success in boosting GPU utilization for autonomous trucking, while Discord detailed its transition from open-source Ray to the Anyscale platform to manage its growing machine learning demands. Cubist/Point72 discussed their use of Ray for model fitting in quant finance, and Coinbase presented their journey of enhancing batch ML processes with Anyscale Job Queues, achieving significant improvements in job stability and cost efficiency. The day also included hands-on workshops on Ray's core features, distributed training, and robotics simulation, offering insights into the future of AI infrastructure.

Training data often resides in different cloud regions than the GPUs used for AI model training, resulting in significant latency due to cross-region data reads. To address this, deploying Alluxio as a compute-side NVMe-based distributed cache with Ray on the Anyscale platform significantly accelerates data access by storing data locally after the first read, eliminating repeated cross-region data transfers. In a benchmark test, using Alluxio with Ray Data to read 1TB of Parquet files resulted in a warm cache read speedup of 20x, reducing the time from over 4,200 seconds to approximately 208 seconds. This solution supports storage-agnostic caching, improving data read performance across multiple cloud providers like AWS, Google Cloud, and Azure, and is particularly beneficial for multi-epoch training and hyperparameter tuning scenarios where datasets are repeatedly accessed. Additionally, the integration does not require changes to existing training code, making it an effective optimization strategy for AI workloads distributed across cloud regions.

Anyscale on Azure, now in public preview, offers enterprises the ability to build and deploy AI workloads at scale within their own Azure environments, leveraging the open-source Ray compute engine. This integration, co-engineered with Microsoft, provides the same governance and operational controls as other Azure services and allows enterprises to use their existing Microsoft Azure Consumption Commitments (MACC) for Anyscale's AI workloads. Anyscale's platform, which includes the Anyscale Runtime, Workspaces, and Agent Skills, is designed to handle GPU-accelerated tasks such as multimodal data processing, distributed training, and large-scale inference, which traditional cloud-native services struggle to support. Companies like Wayve and Xoople are utilizing Anyscale on Azure for complex AI tasks, such as autonomous driving AI and satellite imagery analysis, respectively. The service is seamlessly integrated with Azure's security, governance, and billing frameworks, allowing enterprises to manage AI workloads alongside their existing infrastructure without needing separate contracts.