June 2024 Summaries

The Ray team at Anyscale recently transitioned to weekly releases to keep up with the rapid pace of innovations in AI models, accelerators, and techniques. Prior to this change, they followed a six-week release cycle that introduced noise into determining root stability and reliability issues due to flaky tests, heavy lifting for releases, and customer frustration. To address these issues, the team tackled instability within Ray's codebase by addressing test failures, systematic updates, and cultural shifts. The transition has resulted in enhanced stability with approximately 350+ issues fixed, streamlined releases, improved productivity, but also presents challenges such as frequent upgrades and API stability. To address these challenges, the team is exploring ideas like a Long-Term Support version and managed upgrade services to provide users with a stable foundation and smooth transitions.

This guide provides a step-by-step process for developing and deploying a large language model (LLM) using Anyscale and Ray. The workflow includes data preprocessing, fine-tuning, evaluation, and serving the model. The authors use Ray to distribute the workload across multiple machines, allowing for efficient processing of large datasets. They also provide an example of how to serve the model in production using Anyscale Services, which can be scaled up or down as needed. The guide covers various aspects of the process, including data preprocessing, fine-tuning with different optimization techniques, and serving the model with LoRA adapters. It also discusses the importance of consistency between development and production environments and provides tips for optimizing performance and scalability.

Multi-application support is now available on Ray Serve, allowing different applications to run on the same cluster using separate containers. This feature helps in managing resources more effectively and reduces operational complexity by enabling independent upgrades of each application. With this new "runtime envs as containers" feature, users can specify a different Docker image per team's application, providing clean isolation between apps and unlocking the power of Ray's multi-tenancy capabilities. This enables users to share resources more efficiently and utilize GPU capacity granularly, reducing operational complexity and improving scalability. The feature is still experimental and has some drawbacks, such as startup delays and limitations on supported runtime environment fields. Further development plans include cleaning up the UX around combining containers with other runtime environment fields and adding more scalability testing.

The guide outlines the development of a fashion image retrieval system using the Contrastive Language-Image Pre-training (CLIP) models, Pinecone vector database, and Ray Data for efficient data processing. The application allows users to search for images within a dataset using text or image prompts, leveraging the capabilities of CLIP embeddings to combine textual descriptions with visual data in one space. The system consists of several components, including GradioIngress, Multimodal Similarity Search Service, Image Search Service, Text Search Service, and Pinecone, which work together to perform cross-modal search, reranking, and visualization of results. The application showcases the efficiency of parallelization techniques offered by Ray Data, demonstrates the creation of Pinecone indexes, leverages CLIP models for building a cross-modal retrieval pipeline, optimizes application performance with autoscaling using Ray Serve, develops an intuitive Gradio interface, and provides a practical roadmap for building efficient, scalable, and intuitive applications. The guide is intended to empower developers to replicate the work by providing detailed explanations of each step and resources available on GitHub.