January 2023 Summaries

This tutorial demonstrates how to set up an ML model drift monitoring system for a computer vision project using the open-source library whylogs and WhyLabs. The case study involves monitoring a computer vision classifier trained to distinguish between cats and dogs, with data divided into day batches. Data drift is detected by profiling the data with whylogs and sending the profiles to WhyLabs for monitoring. Human annotation using Toloka's crowdsourcing platform helps confirm whether model performance has been affected by the drift. Comparing model predictions with human annotations in WhyLabs reveals a drop in accuracy, indicating that the model has indeed drifted.

Google BigQuery users can now monitor the quality of their data using whylogs without writing any code. This is the first no-code solution for data monitoring offered by WhyLabs, which started with BigQuery due to its popularity and managed infrastructure. Data quality monitoring is crucial for ensuring that the data used in analytics and machine learning applications is reliable. The integration uses an Apache Beam template published to a public GCS bucket, allowing users to create a Dataflow job that consumes from BigQuery in various ways depending on how it's configured.