November 2025 Summaries

Ray Serve LLM introduces new APIs that facilitate the deployment of advanced serving patterns for sparse mixture-of-experts models, like DeepSeek and Qwen3, using vLLM on the Anyscale platform. The APIs support wide expert parallelism and disaggregated prefill/decode serving, enabling models to achieve high throughput and optimized latency by balancing expert loads and separating processes that handle input prompts from those generating output tokens. By leveraging Ray Serve, developers can build and orchestrate complex model deployments using Pythonic builder patterns, which allow for dynamic scaling, stateful routing, and fault-tolerant orchestration, while maintaining compatibility with Kubernetes environments. This approach reduces the operational burden of coordinating multi-node setups and enhances performance through programmable orchestration, enabling efficient use of resources and maintaining high service level agreements.

Anyscale is showcasing its latest innovations in AI infrastructure at AWS re:Invent 2025, offering attendees opportunities to learn about its advancements and engage with experts. Visitors can explore features like Lineage Tracking, Anyscale Runtime, and the Global Resource Scheduler at Booth #1854, while also having the chance to receive personalized consultations with the Anyscale team. The event includes participation in Intel's booth and executive events, where topics such as scalable AI and ROI are discussed. Additionally, Ray, the open-source framework by Anyscale, is highlighted in AWS-led sessions focusing on distributed AI and scalable LLM deployments. For those unable to attend in person, a virtual hands-on lab is scheduled for December 11.

Ray has introduced a token-based authentication system to enhance security within its framework, addressing a previously missing layer in its open-source offerings. This mechanism ensures that only requests with valid tokens are processed by Ray's internal and external services, thereby offering a secure, out-of-the-box experience without the need for complex setups or external identity systems. The unified token model secures communication across all Ray components and is transparent for developers, ensuring end-to-end protection. While some local communications remain exempt due to their trusted nature, the system provides comprehensive coverage across public and private APIs. Initially available as an opt-out feature to facilitate transition, token-based authentication is set to become the default in future Ray releases, marking a significant advancement in the project's security posture and aligning with a secure-by-default approach for distributed systems.

Ray Serve has introduced several new features to enhance its flexibility and scalability for modern AI inference workloads, accommodating the needs of teams handling multimodal AI tasks. These features include Async Inference, which allows for the safe and efficient management of long-running workloads by integrating asynchronous processing into the serving layer, eliminating the need for additional infrastructure. Custom Request Routing provides precise control over request distribution, enabling domain-specific routing logic that can optimize system performance. Custom Autoscaling allows developers to define scaling policies using custom metrics, offering granular control to balance throughput, cost, and latency, while External Scaling permits programmatic adjustments to replica counts via external data sources. Collectively, these advancements make Ray Serve more adaptable and programmable, streamlining the process of deploying complex AI systems in production environments by ensuring safe execution, improved latency, and optimized resource utilization.

Anyscale and Microsoft have announced a strategic partnership to integrate the open-source distributed compute framework Ray into Azure, creating a new AI compute service co-engineered to meet the needs of enterprises building AI-native platforms. This collaboration aims to address the growing complexity and scale of AI workloads, which have outpaced traditional computing architectures. By leveraging Ray's flexibility and Azure's cloud capabilities, the partnership seeks to enable enterprises to efficiently manage multimodal data pipelines, reinforcement learning, and distributed inference across diverse hardware without the burden of orchestrating complex systems. This service, integrated with Azure Kubernetes Service, provides a unified, secure environment for AI development and deployment, empowering enterprises to innovate with reduced operational overhead. The partnership signifies a shift from API-first to code-first AI development, granting enterprises the control and resources needed to create custom AI models and platforms.

Anyscale has introduced Lineage Tracking, a new feature in its platform aimed at providing comprehensive visibility into distributed AI development workflows, particularly those using Ray. This capability is designed to address the challenges of tracing and debugging in distributed environments by offering a complete overview of the data and models' journey through Anyscale Workspaces, Jobs, and Services. Lineage Tracking features interactive lineage graphs that visually map the connections between datasets, models, and compute resources, facilitating easier experiment reproduction and debugging. It integrates natively with popular tools like Unity Catalog, MLflow, and Weights & Biases, enabling seamless context retention and dependency analysis without the need for custom scripts. Built on the OpenLineage standard, the feature ensures portability and compatibility with various metadata management and tracking systems, allowing teams to trace and reproduce runs efficiently while maintaining governance across AI pipelines.

Ray Data, a scalable data processing framework, has experienced significant growth and adoption since its general availability announcement, driven by evolving demands for handling multimodal data and large AI models. The platform has expanded its capabilities to support high-dimensional datasets such as images and embeddings, requiring specialized formats and inference engines, and has improved structured data operations through enhanced DataFrame APIs and optimized functions like projection and predicate pushdown. Recent updates include features for efficient multimodal data processing, such as improved tensor handling and direct MCAP file reading, as well as enhancements for large model support, including cross-node model parallelism and compatibility with various accelerators. Ray Data 2.51 also introduced new APIs that facilitate vectorized transformations, improving the efficiency of wide operations like joins and shuffles, and optimized parquet reading performance. These developments aim to meet the needs of growing data and AI workloads, emphasizing performance, reliability, and scalability.

Anyscale, in collaboration with Microsoft, has launched a private preview of its first-party offering on Azure to streamline the building, running, and scaling of AI-native workloads using Ray's distributed compute framework. This service allows Azure customers to seamlessly integrate and manage AI workloads within the Azure Portal, leveraging Azure's existing security and compliance frameworks for secure authentication and authorization. Anyscale provides cost-efficient and high-performance processing through the Anyscale Runtime, optimized for enhanced reliability and speed, and supports existing Azure-native services, enabling developers to deploy AI environments quickly without the operational overhead. The integration allows for unified billing and management, offering Azure customers the ability to efficiently execute large-scale AI projects within the secure infrastructure of Azure, thereby accelerating digital transformation initiatives.

Ray Train V2 introduces several enhancements to improve the experience of distributed training, focusing on better usability, reliability, and a cleaner API surface for faster feature development. Key features include asynchronous checkpointing, allowing model checkpoints to be uploaded in a separate CPU thread to maintain high GPU utilization, and asynchronous validation, enabling parallel validation without blocking the training loop. It also introduces a JaxTrainer API for seamless scaling of JAX training on TPUs, providing a single-controller orchestration for greater fault tolerance compared to the traditional multi-controller setup. Additionally, a new local mode allows for streamlined debugging by executing training functions directly in the current process, offering both single-process and multi-process modes. These advancements lay the groundwork for future releases aimed at enhancing fault tolerance, framework integrations, and experiment management.

Ray Summit 2025 highlighted significant updates to the Anyscale Platform, focusing on enhancing developer velocity, cost efficiency, and production resilience. Key announcements included the introduction of Lineage Tracking, which integrates with MLflow, Weights & Biases, and Unity Catalog to provide developers with a clear visualization of data and model workflows, and the Anyscale Runtime, which offers a high-performance, cost-efficient engine for Ray-compatible workloads without requiring code changes. The summit also introduced Anyscale on Azure, a first-party offering that integrates with Azure infrastructure, and improvements in observability with new dashboards for Ray workloads. The Global Resource Scheduler and Multi-Resource Cloud (MRC) were launched to optimize resource utilization and job scheduling across cloud environments, facilitating AI development and deployment. Additionally, the Anyscale Cluster Controller was introduced to simplify cluster management, offering better governance and cost controls, while Custom Workspace Templates aim to streamline developer onboarding with reusable setups. These enhancements aim to improve scalability, performance, and reliability for AI teams using Ray and Anyscale across various cloud platforms.

Anyscale Runtime, powered by the open-source distributed compute framework Ray, aims to enhance the efficiency, cost-effectiveness, and resilience of AI workloads. It addresses the challenges faced by modern AI tasks, which require complex data processing across heterogeneous compute resources, by offering features like job checkpointing, mid-epoch resume, and dynamic memory management to reduce failures and improve stability. Anyscale Runtime enables seamless integration with existing Ray applications, ensuring higher throughput and lower costs across various AI processes such as image batch inference, feature preprocessing, structured data processing, high-throughput serving, and online video processing. Organizations such as Geotab and TripAdvisor have reported significant improvements in throughput, GPU utilization, and cost savings. The runtime's benchmarks demonstrate substantial performance gains, with up to tenfold improvements in various tasks compared to the open-source Ray framework, showcasing its potential as a robust engine for AI workloads.

Ray Direct Transport (RDT) is a new feature in Ray Core designed to enhance the efficiency of GPU data transfers by leveraging high-bandwidth communication technologies such as RDMA and NVLink, particularly for reinforcement learning (RL) applications involving large language models (LLMs). This development addresses the limitations of Ray's CPU-based object store in handling large tensor transfers between GPUs by enabling direct GPU-to-GPU communication, significantly speeding up data transfers by up to 1000 times with minimal code changes. RDT supports various transport libraries like NCCL and NIXL and allows users to specify data dependencies while choosing the most suitable transport method for their setup. In a practical application, RDT can accelerate weights synchronization and data transfer in RL training loops, as demonstrated in a minimal RL example using the Group Relative Policy Optimization (GRPO) algorithm. Currently in its alpha phase, RDT is focused on Ray actors, with ongoing efforts to expand its capabilities and gather user feedback for further enhancements.