August 2023 Summaries

Data drift is a crucial aspect of Machine Learning Operations (MLOps) that refers to the change or variation in input data over time. This can occur due to natural changes, evolving user patterns, or shifts in business environments. Ignoring data drift can lead to decreased model performance and unreliable predictions. To manage this, continuous monitoring is integrated into MLOps pipelines. Tools like whylogs help detect data drift by comparing the distribution of training data with that of production data. Once detected, mitigation strategies such as retraining models or adjusting thresholds can be implemented.

Glassdoor has decreased latency overhead and improved data monitoring by integrating with WhyLabs. The challenge of integration latency was addressed by optimizing the whylogs open-source data logging library, which resulted in a significant reduction of latency overhead from 68 ms to 8 ms. This case study highlights the importance of examining the implementation of functions being integrated into services and demonstrates that ML observability can be lightweight while still providing valuable insights for Glassdoor's ML systems.

The healthcare industry is increasingly relying on artificial intelligence (AI) for disease diagnosis, patient outcome predictions, and overall patient care. With the growing reliance on AI systems, it's important to acknowledge that the likelihood of bad or changing data resulting in incorrect predictions or recommendations also increases. Healthcare companies need to keep an eye on the well-being of their models and the quality of the data used to train and update them to ensure their AI systems are performing as expected. Key reasons for implementing an ML monitoring solution include identifying issues before they become serious problems, improving patient outcomes, reducing costs, and ensuring compliance and regulation.

CB Insights has recognized WhyLabs as one of the top 50 most innovative companies developing generative AI applications and infrastructure. The recognition highlights Model Observability's importance in ensuring the success of LLMs (Large Language Models). As a pioneer in this category, WhyLabs is dedicated to establishing Observability as an essential tool for organizations running AI in production. Their focus has been on creating best practices for operating Generative AI models and providing practitioners with tools to operate models with transparency, rigor, and safety. With the recent launch of LangKit, users can now identify and address various risks within their LLMs, including toxic language, data leakage, hallucinations, and jailbreaks.