September 2023 Summaries

Galileo has developed a new platform called LLM Studio, which helps teams develop and evaluate large language models (LLMs) in hours instead of days. The platform offers three modules: Prompt, Fine-Tune, and Monitor, designed to address the challenges faced by LLM developers today, including prompt engineering, fine-tuning, and observability and monitoring. LLM Studio provides features such as automatic version controls for collaboration, industry-standard metrics, and custom evaluation metrics, as well as a Guardrail Metrics Store to minimize risk and bring more trustworthy applications to market. The platform is built with enterprise needs in mind, offering privacy-first data residency, high customization options, quick onboarding, and exciting enhancements on the way.

There has been tremendous progress in the world of Large Language Models (LLMs), with blockbuster models like GPT3, GPT4, Falcon, MPT, and Llama pushing the state of the art. However, evaluating these models is challenging due to their tendency to hallucinate. To address this issue, companies are developing evaluation metrics that can help them make data-driven decisions without relying solely on human judgment. These metrics include context adherence measures, correctness metrics, log probability-based metrics, prompt perplexity, and safety metrics such as PII, toxicity, tone, sexism, and prompt injection detection. By using these metrics, companies can identify potential issues with their LLMs, optimize their performance, and ensure that they are generating high-quality outputs that meet the needs of their users.