November 2023 Summaries

We are excited to announce that Anyscale's LLM API Offering, including Private Endpoints for self-hosted LLMs, is now available as part of the Anyscale Platform. This marks a significant milestone in our journey towards democratizing access to large language models. With our new endpoint offering, developers can easily integrate open-source embedding models into their applications, such as retrieval-augmented generation (RAG) applications, at an affordable price point of $0.05/MTokens for the gte-large model. We also plan to add more models in the future and invite users to request newer embedding models through a Google form. Additionally, we have extended fine-tuning functionality to our Llama-2 70B model, allowing developers to improve model quality while reducing costs and improving performance. The fine-tuned model can now be used for inference at $1/M tokens. Furthermore, users can now get started with Anyscale Endpoints without a credit card, receiving free credits that can be added to their account later.

Ray has released patches for four of the five reported CVEs (CVE-2023-6019, CVE-2023-6020, CVE-2023-6021, and CVE-2023-48023) in master and will be part of Ray 2.8.1. The remaining one (CVE-2023-48022), related to lack of authentication built into Ray, is a design decision based on how security boundaries are drawn and consistent with best practices. Ray's security boundary is outside the cluster, so it does not consider this bug a vulnerability or even a bug. However, they recognize its potential value in defense-in-depth strategy and plan to implement authentication as a new feature in a future release.

The Generative AI boom is underway, with large language models creating a global understanding of its potential. The platform vendor community is racing to deploy AI workloads, but challenges include defining use cases, executive sponsorship, and AI skills on deployment teams. A significant constraint is the scarcity of specialized compute resources for AI workloads. To address this, we can optimize existing resources or increase their availability. Anyscale is pushing compute efficiency boundaries, while Lambda is expanding access to specialized AI hardware. Anyscale has partnered with Nvidia and validated its integrations with Lambda's offerings, enabling rapid testing and deployment of large language models. This collaboration has provided the LLM developer community with a self-serve way to assess performance and accelerated delivery of new tooling.

Open-source large language models (LLMs) like LLaMA have become viable for production use cases, offering significant cost savings compared to proprietary models like GPT-3. While they may lag behind in terms of output quality, instruction following, and function templates, open-source LLMs can still provide good performance for many applications, especially when fine-tuned for specific tasks. The use of hybrid approaches combining both open source and proprietary LLMs can offer a cost-performance tradeoff. As the community continues to innovate and address limitations, open LLMs are becoming increasingly attractive options for developers and businesses, particularly in scenarios where cost-effectiveness is key.

Anyscale has developed a GitHub bot called "Docu Mentor" that analyzes writing style and grammar in pull requests. The bot uses a large language model (LLM) to provide suggestions for improvement, and can be triggered by comments such as "@docu-mentor run". To build the bot, developers must create a Ray Serve app with a FastAPI API definition, which is then deployed on Anyscale using a YAML file. Once deployed, the bot can be used to analyze PRs and provide feedback in the form of GitHub comments. The bot's performance was evaluated using GPT-4 as an evaluator, and found to be effective in identifying grammatical errors with a high degree of accuracy. To deploy the bot on GitHub, developers must create a new app with the correct permissions and settings, and link it to their repository.

Spotify's journey to building a robust Ray platform with a frictionless developer experience is an inspiring story of innovation and streamlining the machine learning development process. They leveraged Kubernetes, Ray, and custom SDKs to create a user-friendly Cloud Development Environment (CDE) that simplified the development workflow for ML engineers, researchers, and data scientists. Spotify's CDE improved productivity by eliminating environment issues, providing more compute power, and enabling frictionless onboarding for users of diverse backgrounds. Key lessons learned include ensuring availability, performance, and security, allowing for customization and extensibility, and using Kubernetes to leverage its features. Spotify integrated PyTorch support with Ray for scalable training and hyperparameter tuning, driving more ML innovations. Their machine learning platform powers various applications, including personalized content recommendations, search result optimizations, and content discovery, and has an SDK with Ray and PyTorch libraries to standardize common ML tasks.

LinkedIn's AI platform is undergoing a transformation with the help of Ray and Inference Graphs, simplifying complex AI workflows and enabling better integration, resource utilization, and seamless transitions between online and offline inference. The company faced challenges such as managing complex AI workflows, integrating multiple programming languages, and adapting to an ever-evolving AI landscape. LinkedIn's solution involves adopting Ray and the concept of Inference Graphs, which simplify AI workflows, allow for heterogeneous infrastructure, and enable seamless transition between offline and online inference. The platform has seen significant changes in handling AI workloads, with minimal overhead for LLN use cases and optimized serialization and deserialization costs for personalization models. LinkedIn is actively exploring further optimizations to maximize the efficiency of AI workloads.

The results of a blind test conducted by Anyscale showed that Llama 2 70b was slightly outperformed by human legislative interns when summarizing bills, while GPT-4 significantly outperformed both humans and Llama 2 70b. The test involved generating summaries for 28 legal bills from the BillSum dataset and scoring them on a scale of 1 to 5. The analysis revealed that GPT-4's superior performance was likely due to its ability to guess at what the user wanted, as well as its training on legislation, which may have given it access to external sources of information not available to Llama 2 70b. To improve Llama 2 70b's performance, Anyscale modified its prompt to focus on the most important aspects of the bill and used active verbs to describe the bill, similar to GPT-4's approach. The tweaked prompt led to improved summaries with more "summary-like" features, suggesting that using a powerful tool like GPT-4 can help refine and develop less capable learners.

The latest release of Ray 2.8 brings significant enhancements to the ecosystem, focusing on data ingestion and profiling capabilities. The addition of support for reading external data sources like BigQuery and Databricks tables allows for seamless integration with popular data stores, augmenting data ingestion for machine learning training. Ray Data now provides real-time metrics on consumption, including bytes spilled, consumed, allocated, outputted, freed, as well as CPU and GPU usage, enabling better monitoring of operations. Furthermore, the release introduces support for AWS Neuron Core accelerators, providing a range of GPU accelerators to improve heterogeneous training and batch processing performance and efficiency. NVIDIA Nsight System is also natively supported on Ray, allowing for profiling GPU-bound tasks and actors, offering insights into job health and progress. Overall, this release aims to enhance ease of use, performance, and stability while expanding Ray's capabilities with experimental features like AWS Neuron Core accelerators and improved data ingestion functionality.

The field of Artificial Intelligence (AI) and Machine Learning (ML) is rapidly evolving, with foundation models playing a crucial role in transforming various sectors of society. These models serve as the backbone of AI applications, providing a basis for tasks such as text generation, image processing, and video generation. Academia has taken an important role in researching foundation models responsibly, advancing their capabilities through technical innovations, applications, and social responsibility. Researchers have developed new techniques to improve data selection, model architectures, optimization, and alignment, leading to significant speedups in training the models. Foundation models are being applied to domains such as law, healthcare, and robotics, where they can make a significant impact. However, their societal implications require careful consideration, including copyright, privacy, and watermarking. It is essential for organizations involved in AI development to prioritize transparency and responsible practices to ensure the responsible use of foundation models and unlock their full potential.

Anyscale Endpoints (LLM API Offering) and Private Endpoints are now available as part of the Anyscale Platform. The release of LLMPerf, an open source project for benchmarking LLMs, aims to make claims about LLM performance reproducible by standardizing on key metrics such as latency, throughput, and cost. The benchmarks show that Fireworks.ai and Anyscale Endpoints are viable alternatives, with Anyscale being 15% cheaper and 17% faster than Fireworks in typical workloads. However, the choice of LLM depends on the specific application, with ultra-low latency applications potentially benefiting from Perplexity's open beta, while large workloads may favor Anyscale or Fireworks. The LLMPerf benchmarking tool is available for download and aims to improve transparency and reproducibility in comparing LLM outputs.