September 2023 Summaries

Ray Serve and Anyscale Services are now generally available, offering a better way to serve machine learning models that is flexible, performant, and scalable. These solutions aim to solve common challenges in AI application development, such as improving time to market, reducing cost, and ensuring production reliability. Ray Serve provides simplicity, flexibility, and scaling, while Anyscale Services manages deployment infrastructure and integrations, ensuring reliable production deployments with zero-downtime upgrades and canary rollouts. The combination of Ray Serve on Anyscale Services optimizes the full serving stack across model, application, and hardware layers, making it a future-proof solution for AI applications. With its flexibility, scalability, and performance, Ray Serve has already seen significant adoption in various industries, including Ant Group and Samsara, which have improved their production ML pipeline performance and reduced costs. Anyscale Services also supports heterogeneous hardware support, model multiplexing, and request batching, providing cost reductions of 2-3x and better GPU availability. The solution is designed to meet the growing demand for AI applications and provides a managed and production-ready platform for building and deploying AI applications.

The latest release of Ray 2.7 brings significant stability improvements, enhancements to libraries and KubeRay for Kubernetes, and introduces new features such as RayLLM for serving open-source large language models (LLMs) with Ray Serve. The update simplifies APIs in Ray Train for general availability, stabilizes and enhances Ray Serve and KubeRay, and adds support for various accelerator devices including TPUs, Trainium, and Inferentia. Additionally, Ray Data has improved performance with features such as zero-copy fusion for map operators and multithreaded file reading. The release also includes a unified DeploymentHandle API, gRPC ingress support, websocket support with FastAPI, streaming responses, batch requests, model multiplexing, and multi-app support. The update is part of Ray's efforts to simplify the number of concepts that users need to learn about and reduce friction for new machine learning practitioners to quickly adopt Ray Train for distributed training at scale.

Ray Data provides fast, flexible, and scalable data loading capabilities for ML pipelines, overcoming common challenges such as GPU utilization and memory usage. It leverages Ray Core's distributed execution to scale out data preprocessing tasks across multiple GPUs, heterogeneous clusters, and cloud storage. With features like streaming execution, caching, auto-partitioning, and recovery from transient errors, Ray Data offers unmatched flexibility and scalability in multi-node settings. By comparing its performance with popular open-source data loaders, such as PyTorch DataLoader and tf.data, Ray Data demonstrates its ability to handle large-scale image data preprocessing tasks efficiently. Its active development ensures that it will continue to improve its performance and features, making it a valuable tool for developers and researchers in the ML community.

The Anyscale platform has evolved to accelerate the development and deployment of Generative AI (GenAI) and Large Language Model (LLM) applications, allowing developers to focus on innovation while simplifying infrastructure management. Instance startup times have been significantly improved through optimizations, enabling faster autoscaling and quicker reaction to traffic. The platform optimizes compute availability by intelligently determining the most suitable instances across clouds, regions, zones, and instance types, particularly for GPU-intensive workloads. Anyscale also provides a flexible approach to balancing cost and reliability, utilizing spot instances with fallbacks to ensure availability without compromising reliability. Additionally, the platform offers advanced security controls, including finer-grained access controls, and provides fast, scalable storage solutions through managed NFS cloud storage and Non-Volatile Memory Express (NVMe) support. By eliminating the need to manage infrastructure, Anyscale enables teams to concentrate solely on innovation and delivery, making it an attractive solution for organizations developing GenAI and LLM applications.

Anyscale has partnered with Meta to provide access to Llama-2 models via Anyscale Endpoints, an OpenAI-compatible LLM inference API for open models. This collaboration aims to bring high-quality cost-efficient LLM inference to a broad range of application developers. The Llama-2 models have already gained popularity due to their performance on various benchmarks and the emergence of an ecosystem across academia and industry. Anyscale's Endpoints enable developers to easily switch between closed models and open models, offering flexibility and efficiency in AI application development. With this partnership, cost efficiency is expected to play a crucial role in improving model quality while maintaining speed and cost efficiency.

We compare full-parameter fine-tuning with LoRA (Low-Rank Adaptation of Large Language Models) and explore their strengths and weaknesses. We train Llama 2 models on the same three real-world use cases as in our previous blog post to provide a baseline for task-specific performance, hardware requirements, and cost of training. The results show that using LoRA involves a trade-off between serving efficiency and model quality, which varies according to the specific task at hand. Additionally, we offer insights into how to stabilize training with LoRA through intelligent prompting techniques. We further demonstrate that adopting a lower learning rate can enhance the reliability of the resulting model checkpoints. Our experiments show that LoRA fine-tuned models are only slightly worse than full-parameter fine-tuned models for tasks like generating SQL queries or text-based functional representations, but fall short in mathematical reasoning tasks. By leveraging LoRA's efficiency in memory and serving, we can deploy multiple fine-tuned models simultaneously while reducing storage requirements, making it a promising alternative to full-parameter fine-tuning, especially when operating on cheaper lower-memory instances or with larger context lengths.