September 2025 Summaries

Ray Serve, a scalable model serving library built on Ray, has introduced a custom request routing feature, the PrefixCacheAffinityRouter, which significantly reduces latency in large language model (LLM) inference, particularly for models like Deepseek-R1 and Kimi K2. By utilizing a prefix cache that stores computed key-value vectors from previous requests' attention computations, the router effectively routes requests sharing a common prefix to the same replica, optimizing cache hits and reducing GPU cycle waste. This approach enhances performance, achieving a 60% reduction in time-to-first-token (TTFT) and over 40% improvement in end-to-end throughput, particularly benefiting large Mixture of Experts models that require efficient data parallel attention and expert parallel sharding. The new routing strategy was benchmarked using the PrefixRepetitionDataset, demonstrating improved throughput and constant cache hit rates even as the number of replicas scales, compared to the traditional "Power of Two Choices" routing strategy.

The announcement details the introduction of a persistent Ray Task Dashboard on Anyscale, designed to improve the monitoring and debugging of Ray workloads by providing a scalable solution that maintains visibility even after cluster shutdowns. This development leverages the new Ray Event Export framework available from Ray 2.49, addressing previous limitations in task observability such as the 10,000 task limit and non-persistence of data. The dashboard offers a comprehensive view of task performance, including high-level summaries, aggregated views, and individual task analyses, all updated in near real-time. Built with a focus on data privacy and cost efficiency, the system utilizes a dedicated cluster within the customer's cloud account, ensuring that sensitive data remains secure while providing a responsive user experience. The Ray Event Export framework facilitates detailed task-level visibility by emitting structured events, enabling users to build custom dashboards for individual needs. Future plans include expanding the event export to cover more Ray entities and enhancing other Ray dashboards with persistent task data, ultimately aiming to automate issue detection and workload optimization.

The blog post discusses the challenges and solutions involved in running massively parallel agentic simulations with Ray, a Python-based framework. It highlights the importance of such simulations in various use cases, including evaluating and improving large language models (LLMs), iterating on datasets, and running reinforcement learning (RL) training. The authors describe how Ray addresses issues like agent isolation, scaling model inference, and using custom models, enabling fast experimentation and scaling without rate limits. Ray's capabilities are demonstrated through examples such as running evaluations, iterating on the cpython issues dataset, and integrating with RL libraries like SkyRL. The post also delves into different methods of isolating simulations, including using containers, processes, and virtual machines, and presents the mini-swe-agent as a flexible tool for executing agent actions. Overall, the blog emphasizes the flexibility and scalability of Ray for handling large-scale, distributed agentic workloads efficiently.

Anyscale has announced the availability of its platform on the Microsoft Azure Marketplace, facilitating the deployment of AI workloads using Ray on Azure Kubernetes Service (AKS). This integration offers enterprises streamlined procurement and billing processes, the ability to use existing Azure commitments, and flexible commercial terms. Anyscale enhances AI development with purpose-built tools for Ray, including multi-node IDEs and built-in dashboards, and reduces operational costs through managed and elastic clusters. The platform also features RayTurbo for optimizing hardware utilization, helping organizations achieve better ROI on AI investments. Enterprises can access expertise from the creators of Ray for support and training, ensuring efficient deployment and scalability of AI projects. The announcement also highlights the upcoming Ray Summit 2025, where participants can explore real-world applications and product updates.