May 2023 Summaries

Aviary is an open source project that simplifies the self-hosted serving of multiple LLM models efficiently, addressing issues such as cost, latency, transparency, deployment flexibility, data control, and customization. The project leverages Ray Serve to provide a highly flexible serving framework for scalable AI, offering pre-configured LLMs, acceleration approaches, simplified deployment, and autoscaling support. Aviary aims to bring these capabilities together in a convenient way for users, with plans for future expansion and community contributions. A hosted version of Aviary will also be offered, providing additional features around deployment and management, while existing Anyscale customers can access it at no additional charge via their workspaces solution.

The SewerAI team developed a computer vision model to improve the inspection process of municipal sewer infrastructure, reducing costs and improving safety. They used Anyscale's Ray project to scale their machine learning workloads, achieving a 3x speedup in batch inference, reducing GPU utilization from 25% to over 95%, and saving over 1 FTE. The solution enables municipalities to map, monitor, and maintain critical wastewater infrastructure more efficiently, with improved accuracy and reduced operating costs.

A study was conducted to provide useful numbers for LLM developers, similar to the "Numbers every Engineer should know" document at Google. The cost of using Large Language Models (LLMs) can be significant, with prices ranging from $0.2c to 6c per 1000 tokens for GPT-3.5-Turbo and GPT-4 respectively. However, it is more cost-effective to use a smaller model, such as GPT-3.5-Turbo, which is roughly 50 times cheaper than GPT-4. Fine-tuning a pre-trained model can also be a cost-effective option, with costs ranging from $7 for fine-tuning a 6B parameter model to $40 for fine-tuning the entire works of Shakespeare. The study highlights the importance of understanding GPU memory and the need for efficient use of resources when working with LLMs.

This blog post discusses three methods of batch inference in Ray: low-level using Ray Actors, high-level using Ray Data streaming, and a combination of both. It explains how to parallelize batch inference on a NYC taxi data model using Ray 2.4, including creating replicas of the trained model as actors, feeding data into these actors in parallel, and retrieving inference results. The post also introduces the ActorPool utility, which simplifies task management, and highlights its unoptimized aspects. Additionally, it discusses how to use the Ray Data library for batch inference, which automates common performance optimizations such as dynamic autoscaling, automatic batching, and pipelining of data, parallelizing data fetching and preprocessing, and managing the actor pool used for inference. The post concludes that Ray Data offers a more expressive and intuitive API for batch inference at scale, while still being layered on top of the underlying Ray Core primitives.

Ray Data provides streaming execution for large-scale batch inference workloads, offering improved performance on heterogeneous clusters with both CPU and GPU devices. This allows for pipelined execution across an entire cluster, avoiding unnecessary overheads associated with bulk synchronous parallel frameworks. By leveraging end-to-end pipelining, Ray Data can handle demanding use cases such as video decoding, annotation, and classification, while also providing optimizations like memory stability, data locality, and fault tolerance to ensure seamless execution. The streaming backend is fully backwards compatible with the existing API, allowing users to transform datasets lazily with map operations and support shuffle operations, caching / materialization in memory, and more. Early users are taking advantage of Ray Data streaming to create efficient large-scale inference pipelines over unstructured data, including video and audio data.

This year's Ray Summit promises to be an exciting event for the AI/ML community, bringing together developers and practitioners to delve into various aspects of the Ray platform. The program features a diverse range of sessions covering topics such as developer-focused Ray-related sessions, applications across different industries, practical deployment strategies, Generative AI, and more. Keynote speakers include luminaries from companies like Uber, Niantic, Inc., OpenAI, and LinkedIn, making it an ideal event for learning about cutting-edge AI/ML platforms and applications built with Ray. The summit also includes community activities and a happy hour, providing opportunities to network with peers and learn from the best in the field.

This blog post builds upon the previous part of a LangChain series to create a self-hosted LLM question-answering service using Ray and StableLM. The system queries search results from a semantic search engine, generates a prompt with the results, and feeds it to an LLM to generate an answer. The code uses a template to specify the LLM's behavior, including setting its "personality" and providing context for the question being asked. The chain is created using LangChain, which provides a powerful combination of Ray and StableLM capabilities. The blog post includes examples of how to use the system with Weights and Biases tracing and logging.

Offline batch inference is a critical workload for many AI products, and addressing its challenges requires a solution that can manage compute infrastructure, optimize resource utilization, transfer data efficiently, and provide a user-friendly experience. Ray Data emerges as the best practical solution for offline batch inference, outperforming AWS SageMaker Batch Transform and Apache Spark by up to 17x and 2x respectively in image classification benchmarks. Its ability to scale effectively to terabyte-sized datasets, stream data through CPU and GPU stages, and support heterogeneous clusters makes it an ideal choice for deep learning workloads. Ray Data's Python native programming model, native support for multi-dimensional tensors, and autoscaling capabilities further enhance its performance and user experience.

This blog post discusses how to turbocharge embeddings using the Ray framework for distributed computing and data processing. The authors show how to scale out document embedding generation to parallelize across 20 GPUs, leveraging Ray Data, a distributed data processing system part of the Ray framework. They use LangChain to load and embed documents, split text into chunks, and store embeddings in a FAISS vector store. By using Ray Data, they can generate and store embeddings for 2,000 PDF documents from cloud storage in under 4 minutes and in less than 100 lines of code. The authors demonstrate how to use Ray clusters on AWS or other cloud providers and explore the potential of combining a vector database with an LLM to create a fact-based question answering service.