July 2024 Summaries

Today, I'm thrilled to welcome Keerti Melkote as our new CEO, a visionary who founded Aruba Networks in his garage in 2001 and led it through monumental growth. We started Ray and Anyscale with the bet that AI will transform every business and industry, and now we're seeing this bet play out as companies like Pinterest, Amazon, and Uber adopt distributed computing for their AI workloads. Our business has transformed over the past year, quadrupling revenue and growing 6x year over year, with customers running workloads on hundreds of thousands of CPU cores and many thousands of GPUs, spinning up clusters in minutes. We're on the cusp of a new era where data-intensive workloads will become far more compute intensive, AI will be used for unstructured data preparation, and inference complexity will explode. This is an incredibly exciting time for Anyscale, and we're hiring across the board to build the foundational infrastructure powering AI.

A comprehensive solution for improving a legacy search system over multi-modal data using Anyscale and MongoDB was presented. The solution consists of a scalable, multi-modal data indexing pipeline that performs complex tasks like batch inference, vector embedding generation, and inserting data into a search index. A performant hybrid search backend is also implemented that combines lexical text matching with semantic search capabilities. Additionally, a simple user interface for interacting with the search backend was created. The solution utilizes Anyscale platform as the AI compute platform and MongoDB cloud as the central data repository. Enterprises dealing with large volumes of multi-modal data often require robust search systems to address limitations such as inadequate support for unstructured data and dependence on data quality and relevance.

As part of Anyscale's offerings, Job Queues enable multiple Ray Jobs to be executed on a shared cluster for batch "offline" workloads, streamlining job scheduling and optimizing resource allocation by allowing better utilization of cluster resources. With Job Queues, users can simplify cluster management, governance, and control, as well as take advantage of dynamic scaling, full lifecycle management, and complete observability. The service also supports advanced prioritization algorithms and offers two ways to run Ray Job-based workloads: Anyscale Jobs for dedicated clusters and Job Queues for shared cluster utilization. By leveraging Job Queues, users can save on cluster re-provisioning times, reduce operational overhead, and optimize resource allocation, making it easier to manage batch "offline" workloads.

Running ML training jobs on a cluster of GPU nodes is essential for handling large datasets and models, but it also introduces risks due to failures during training. Anyscale's elastic training feature allows practitioners to train models in reasonable time frames while ensuring continuous execution despite hardware failures or node preemptions, avoiding idle or wasted time. With this feature, users can configure jobs to run on spot instances, which can reduce costs by up to 60%, and automatically scale up when more nodes become available, maintaining the largest possible cluster for timely results. Implementing elastic training in Anyscale requires minimal code changes, allowing developers to adapt their existing code with a simple change in scaling configuration.

We recently collaborated with the LMSys team to develop an open-source LLM Router that delivers high-quality LLM applications at half the cost by dynamically selecting between high-performance proprietary LLMs and cost-effective open-source models. The LLM classifier directs "simple" queries to Mixtral-8x7B, maintaining high overall response quality while significantly reducing costs. Anyscale and LMSys LLMRouter sets a standard for quality and cost savings, achieving higher quality with lower costs compared to public LLM routing systems. The router is now available as an Anyscale Workspace Template, simplifying training and deployment with built-in observability and debugging tools.

To address the limitations of traditional log viewing for Ray, a new unified log viewer was developed to enhance user experience and streamline log management. The system simplifies the interface, integrates a scalable centralized logging system, and provides continuous persistent access to logs even when the cluster is terminated. It also offers features such as searchable attributes, filtering by component or log level, and filter by specific time range, allowing users to quickly find and resolve errors. The unified log viewer has been tested on large clusters with 100GB+ of data and caches results for speed, making it a valuable tool for debugging and optimizing Ray applications.

Ray has traditionally used unstructured logs due to Python's built-in logging library, resulting in slow development time and integration challenges with existing log viewing solutions. To address these issues, Anyscale has introduced structured logging, which standardizes log entries in a JSON format, making them easier to handle and analyze. This enables seamless integration with tools like the ELK Stack or Datadog, and facilitates better indexing and querying of logs. Additionally, the new Log Viewer provides enhanced functionality, including automatically enhanced structured logging, centralized log viewing, scalability, filtering, and context, making application monitoring and debugging more efficient and effective.

Ray Serve, a scalable model serving library built on Ray, helps manage increased traffic but struggles to scale down once traffic abates, leading to resource fragmentation and underutilized resources. This is where Anyscale's new Replica Compaction feature comes in, optimizing resource usage for online inference and model serving by automatically migrating replicas into fewer nodes to reduce costs. With Replica Compaction, Anyscale can detect when a deployment is downscaled and migrate excess replicas into a single node, reducing instance seconds and cost savings. The feature has shown significant efficiency improvements, with an average efficiency gain of ~10% on high-end GPUs like A100s and H100s, translating to substantial cost savings advantages, especially in less scaled scenarios where costs can be reduced by 50% or more.

This summary provides an overview of the text, highlighting key points about building a novel routing framework for Large Language Models (LLMs) using human preference data. The framework directs simple queries to more cost-effective models while maintaining high response quality. The tutorial covers every step from data labeling and fine-tuning LLMs to offline evaluation and conducting offline evaluations on standard benchmarks. It also discusses the importance of balancing the dataset and optimizing inference speed. The final section evaluates the performance of the router against a random router on GSM8K, demonstrating its effectiveness in out-of-domain generalization.