July 2025 Summaries

KubeRay v1.4 is the latest release from the KubeRay team, enhancing the deployment of Ray on Kubernetes with new features such as the KubeRay API Server V2 and Ray Autoscaler V2, both of which improve reliability and observability. The update introduces Service Level Indicator (SLI) metrics, offering insights into Ray cluster health and performance, and a new experimental KubeRay Dashboard for visualizing resources via a web interface. The release addresses challenges in transitioning from Proof of Concept (POC) to production environments by streamlining interfaces for data scientists, simplifying operations, and facilitating dynamic workload management. Additional improvements include support for gang scheduling with scheduler plugins, streamlined Python package management with uv, and Helm chart unit testing to ensure deployment reliability. The update also highlights the integration of large language model (LLM) workloads, allowing users to deploy and manage distributed GPU resources efficiently. The KubeRay community is encouraged to provide feedback and participate in ongoing development efforts.

Tencent's Weixin team has successfully integrated the open-source distributed computing engine Ray into its AI infrastructure to address the technical challenges of deploying large-scale AI applications. By combining Ray with Kubernetes, Weixin has developed AstraRay, a high-performance AI compute platform that efficiently manages resource-intensive tasks like OCR, which requires over a million CPU cores. Ray’s simplicity, robust ecosystem, and ability to scale from local development to large clusters make it an attractive choice for AI computing, enabling Weixin to streamline application deployment, reduce costs, and improve resource utilization. AstraRay's architecture leverages a shared-state scheduling system called Starlink to handle millions of nodes and heterogeneous resources, enhancing reliability and reducing complexity. These innovations have allowed Weixin to support ultra-large-scale AI workloads while maintaining high responsiveness and low costs, preparing the platform for future AI application developments.

Reinforcement learning (RL) is increasingly crucial for developing large language models (LLMs), extending beyond traditional reinforcement learning from human feedback (RLHF) to include verifiable rewards, especially as high-quality pre-training data becomes scarce. Recent advancements highlight this approach's success, exemplified by OpenAI's reasoning models and DeepSeek R1 models. The field is rapidly evolving with open-source RL libraries that reflect diverse design philosophies and optimization strategies. These libraries, including TRL, Verl, OpenRLHF, RAGEN, AReaL, Verifiers, ROLL, NeMo-RL, and SkyRL, offer various features tailored for different RL use cases, such as RLHF, reasoning, and agentic RL, and are assessed based on their flexibility, scalability, and design components like the generator and trainer. The analysis conducted aims to guide researchers and practitioners in selecting suitable tools by providing insights into each library's strengths, weaknesses, and use cases. The choice of RL library depends on specific user requirements, whether focused on performance, flexibility, or the ability to handle multi-turn interactions within environments.