December 2023 Summaries

The Anyscale Platform has made its Anyscale Endpoints (LLM API Offering) and Private Endpoints (self-hosted LLMs) available, building on the introduction of LLMPerf, an open source tool for bringing reproducibility and clarity to LLM performance benchmarks. The LLMPerf Leaderboard is a public dashboard that highlights the performance of LLM inference providers in the market, providing clear, comparative insights into top providers and driving innovation and excellence in language model inference. The latest version of LLMPerf brings significant updates with expanded metrics, customizable benchmarking parameters, and new testing modes, offering unprecedented insights into performance, accuracy, and reliability of various LLM products.

Anyscale Endpoints is a new LLM API offering that provides a wide range of capabilities to empower developers to build their applications. It enables function calling, interaction with external tools, APIs, databases, and other LLMs, and includes features like JSON mode for structured output data. The platform also supports Meta's newest model Llama Guard, which provides content moderation and guardrails to ensure safe prompts and responses. Additionally, Anyscale Endpoints is now available as part of the Anyscale Platform, offering a more comprehensive solution for developers building applications with LLMs.

Portkey and Anyscale Endpoints offer a comprehensive LLMOps stack for developing AI applications with open-source Large Language Models (LLMs). By combining these two tools, developers can create a robust toolkit that combines reliability, security, fine-tuning, and observability. The combination of Portkey's observability layer, security and compliance protocols, and model management & improvement suite with Anyscale Endpoints' private deployment, fine-tuning, and state-of-the-art open LLMs provides a powerful toolkit for developers to build AI applications quickly and efficiently.

The Anyscale platform has introduced JSON mode and function calling capabilities on its LLM API offering, allowing users to tailor outputs to specific schema requirements and enable models to use APIs effectively. The Mistral-7B model is the first to support these features in preview, with plans to extend them to additional models soon. Function calling enables models to select the right function and arguments for a given task, while JSON mode ensures that outputs adhere to valid JSON formats. A test dataset has been created to evaluate the performance of this feature across multiple models, including Mistral-7B, which performs similarly to gpt-3.5-turbo in single-turn interactions. The platform is now available as part of the Anyscale Platform, with users able to try out these features with Anyscale Endpoints today.