March 2024 Summaries

The `Ray` open source cluster has been found vulnerable to malicious exploitation due to potential misconfiguration, but Anyscale-hosted clusters are not affected. To help users verify their cluster configurations and avoid accidental exposure, a tool is being provided by the developers, which includes a client-side script and server-side code that can be used to check for open ports in Ray clusters. The tool will return information on whether or not ports are open, but it does not validate what is running on those ports, so further investigation may be necessary to determine if an open port poses a security risk. The tool has already been pre-configured with defaults that can reach out to a server for verification, and its source code is available under Apache2. Anyscale will also host the tool for community testing, and it will be included in Ray 2.11, expected in April.

Ray Serve is a scalable model-serving library built on top of Ray for building end-to-end AI applications, providing a simple Python API for serving deep learning neural networks and arbitrary business logic. The integration with NVIDIA Triton Inference Server software and the NVIDIA TensorRT-LLM library aims to optimize model inference and reduce GPU costs. Anyscale is teaming up with NVIDIA to combine developer productivity with cutting-edge optimizations, enabling faster deployment of AI applications to production. RayLLM is an LLM-serving solution built on top of Ray Serve, providing pre-configured open-source LLMs and a fully OpenAI-compatible API. Triton Inference Server supports various deep learning frameworks and provides optimizations that accelerate inference on GPUs and CPUs. The partnership allows developers to leverage advanced inference serving capabilities, improve model performance, and simplify AI development with Python.