April 2025 Summaries

Google Cloud and Anyscale have partnered to integrate Anyscale RayTurbo, a high-performance runtime for Ray, with Google Kubernetes Engine (GKE), creating a unified platform that acts as a distributed operating system for AI. This collaboration simplifies the management of AI workloads by combining Ray's Python-native distributed computing parallelism with GKE's robust container and workload orchestration. The integration is expected to boost performance and efficiency in AI development and production, allowing developers to accelerate model experimentation and launch optimized clusters on GKE with ease. With this partnership, Google Cloud and Anyscale aim to empower developers and platform engineers to tackle sophisticated AI projects with unparalleled performance and flexibility, accelerating AI innovation and reducing costs.

Ray Data LLM provides APIs for offline batch inference with LLMs within existing Ray Data pipelines, while Ray Serve LLM offers APIs for deploying LLMs for online inference in Ray Serve applications. Both modules offer first-class integration for vLLM and OpenAI compatible endpoints, addressing common developer pains around batch inference, such as launching multiple online inference servers and proxying/load balancing utilities to maximize throughput. Ray Data LLM simplifies the usage of LLMs within existing data pipelines by providing a Processor object that can be called on a Ray Data Dataset, while Ray Serve LLM allows users to deploy multiple LLM models together with a familiar Ray Serve API, offering features like automatic scaling and load balancing, unified multi-node multi-model deployment, OpenAI compatibility, and composable multi-model LLM pipelines.