June 2023 Summaries

LlamaIndex and Ray are used to build a query engine to answer questions about Ray itself by utilizing its documentation and blog posts. LlamaIndex provides a data framework for building LLM applications, while Ray offers a scalable AI framework that can be used to accelerate ingest, inference, pretraining, and deployment of the query capabilities into the cloud. The application uses parallel processing with Ray's `flat_map` method to process multiple input files simultaneously, reducing computation time and improving hardware utilization. The application then stores the processed data in an index within LlamaIndex, which can be used for downstream LLM retrieval and querying. Ray Serve is used to deploy the application into production, allowing users to seamlessly query the application with questions about Ray. The application demonstrates how to build a powerful query module over data using LlamaIndex + Ray, enabling users to effortlessly ask questions and synthesize insights about Ray across disparate data sources.

The new multi-GPU training stack in RLlib allows developers to efficiently scale their compute resources, achieving up to 1.7x infrastructure cost savings by leveraging distributed training across multiple compute nodes and GPUs. This approach enables the utilization of smaller instances from cloud providers, reducing costs for unused compute resources. By using this stack, developers can optimize resource allocation and significantly reduce expenses while achieving desired performance for their experiments. The multi-GPU training is available in Ray 2.5 and can be enabled by setting specific flags in the AlgorithmConfig for algorithms like PPO, APPO, and IMPALA.

Large language models (LLMs) dominate compute cost for most real-world applications due to their large GPU memory footprint and compute cost. However, traditional batching policies are inefficient, especially with varying sequence lengths. Continuous batching, also known as dynamic batching or iteration-level scheduling, is proposed as a solution. By leveraging vLLM, users can achieve 23x LLM inference throughput while reducing p50 latency. The continuous batching framework improves memory efficiency by allowing for dynamic allocation of GPU memory and reducing wastage. This approach outperforms traditional static batching and improves both throughput and latency. Continuous batching is particularly effective with high variance in sequence lengths, and its performance gap widens when combined with further optimizations such as iteration-level scheduling and advanced memory management techniques like PagedAttention. The continuous batching framework has been implemented in vLLM, Hugging Face's text-generation-inference, and Ray Serve, demonstrating its potential to significantly improve LLM inference efficiency.

The Anyscale Databricks connector enables easy data transfer between Databricks clusters and Anyscale hosted Ray clusters, facilitating machine learning workflows by allowing entire pipelines to be executed within a single Python script. This connector simplifies access to data, improves data security and governance, and enhances scalability while reducing job run times and costs. By leveraging the power of Ray and its integrations with AI frameworks and libraries, users can unlock the latest AI innovations, such as pretrained Hugging Face language models, and develop machine learning workloads in a unified experience within Anyscale Workspaces.

The latest release of Ray 2.5 introduces several key features and enhancements across the Ray ecosystem, including support for training large language models (LLMs) with Ray Train, serving LLMs with Ray Serve, and a multi-GPU learner stack in RLlib for cost-efficient and scalable reinforcement learning agent training. The release also addresses bottlenecks in RLlib agent training by introducing a new multi-node, multi-GPU training stack that reduces costs by 1.7x. Additionally, Ray Serve now provides streaming responses to HTTP input requests, enhancing user experience, and supports multiplexing among replicas of dissimilar-shaped model architectures for efficient deployment of multiple models. The release also improves the usability of Ray Data for batch inference, with features such as a strict mode that requires schemas for all datasets and standalone Python objects are no longer supported. Overall, the Ray 2.5 release aims to improve ease of use, performance, and stability across the Ray ecosystem.