May 2023 Summaries

Machine learning (ML) models are crucial for modern businesses, but monitoring their performance is essential to ensure they maintain expected outcomes. ML monitoring involves tracking the model's performance metrics, data quality, and overall application health. This article discusses five ways to implement ML monitoring in production: detecting data drift, monitoring models for concept drift and performance, checking ML pipelines for data quality, monitoring AI explainability, and ensuring bias and fairness. By implementing a robust monitoring system, businesses can optimize their operations, reduce costs, and mitigate risks, ultimately leading to better outcomes for both businesses and customers.

WhyLabs has released the private beta version of LangKit, a purpose-built monitoring solution for Language Learning Models (LLMs). The adoption of proprietary LLMs presents unique observability challenges. LangKit allows continuous evaluation of LLM-powered applications using essential metrics extracted from prompts, responses, and user interactions. It helps detect issues across quality, sentiment, governance, and security aspects. Unlike other monitoring tools, WhyLabs doesn't store user input or model output but gathers all essential metrics about prompt, response data, and user interaction data as it flows through the model. LangKit effortlessly scales from recording just a handful of sample prompts and responses to handling millions of LLM interactions per hour.