July 2023 Summaries

Hugging Face has partnered with LangKit to provide a solution for observability of large language models (LLMs). By using LangKit's open-source text metrics toolkit, developers can monitor and evaluate the behavior of Hugging Face LLMs. This integration allows users to implement guardrails, evaluations, and observability in their applications. The process involves installing Hugging Face transformers and LangKit, creating a GPT model function, generating example responses, and extracting language metrics using LangKit. These metrics can then be viewed and analyzed in the WhyLabs Observability Platform for monitoring LLMs over time.

This article discusses seven ways to monitor the behavior of Large Language Models (LLMs) using LangKit and WhyLabs. The focus is on tracking the evolution of LLMs, such as ChatGPT, which have revolutionized Natural Language Processing with their ability to generate coherent human-like text. The article covers various metrics like ROUGE, bias, text quality, semantic similarity, regex patterns, refusals, and toxicity and sentiment analysis. It also provides a detailed example of how these metrics can be calculated for ChatGPT's responses over 35 days using the ELITE5 dataset. The monitoring process involves generating whylogs profiles for each day's data and uploading them to the WhyLabs observability platform. The article concludes by analyzing the changes in LLM behavior over time, highlighting improvements in various metrics after a significant upgrade on March 23.

This blog post discusses the importance of safeguarding and monitoring large language model (LLM) applications to prevent potential issues such as toxic prompts and responses or the presence of sensitive content. It explores three key aspects: content moderation, message auditing, and monitoring and observability. The implementation uses whylogs, LangKit, and WhyLabs tools to calculate and collect LLM-relevant text-based metrics for continuous monitoring. By incorporating these techniques, developers can ensure that prompts and responses adhere to predefined guidelines and avoid potential issues associated with LLMs.

This blog post discusses the importance of monitoring large language models (LLMs) and how to get started with monitoring a LangChain application using LangKit and WhyLabs. The article highlights various metrics that can be tracked for LLM usage and performance, such as response relevance, sentiment, jailbreak similarity, topic, and toxicity. It also provides an example of how to use LangKit with Langchain and OpenAI for LLM monitoring, focusing on tracking sentiment changes between prompts and responses. The post concludes by emphasizing the significance of monitoring large language models in production and suggests other relevant signals that can be monitored using LangKit.

In the latest issue of Robust & Responsible AI Newsletter, various topics related to large language models (LLMs) are covered. Databricks acquires MosaicML, showcasing its commitment to democratizing AI and reinforcing its Lakehouse platform as a leading environment for building generative AI and LLMs. Additionally, the article discusses the recent launch of LangKit by WhyLabs, an open-source library designed to detect and prevent malicious prompts, toxicity, hallucinations, and jailbreak attempts in LLMs. The newsletter also highlights various resources such as on-demand sessions from Snowflake Summit and Databricks' Data + AI Summit, a workshop on ML monitoring, and an upcoming event on combining the power of LLMs with computer vision. Furthermore, it provides insights into open-source projects like BentoML's OpenLLM and FlyteCallback for Hugging Face's Trainer.

This week's WhyLabs update covers monitoring LangChain applications, the use of UDFs in whylogs, best practices for monitoring large language models (LLMs), and more. The introduction of ML monitoring to LangChain LLM applications can be achieved with just a few extra lines of code using the WhyLabs callback. This allows prompts and response metrics to be logged in WhyLabs for EDA and anomaly detection. Additionally, custom metrics can provide valuable insights and enable specific monitoring through UDFs in whylogs. The latest blog posts discuss best practices for monitoring LLMs and the introduction of UDFs in whylogs. Upcoming events include an introduction to LLM monitoring in production with LangKit & WhyLabs, MLOps happy hour, and more.