August 2022 Summaries

Whylogs is an open source tool for data logging that enables users to capture different aspects of how their data and models look through snapshots called data profiles. These profiles are mergeable and efficient, allowing users to efficiently answer questions about their datasets without worrying too much about compute costs or storage. By creating lightweight profiles, whylogs can calculate statistical information on datasets quickly and create light representations of the data while still maintaining accuracy. The tool is incredibly efficient even as datasets scale to larger sizes, preventing storage issues and ensuring quick profiling times.

Data quality issues can be challenging for applications dealing with large amounts of data. Schema validation is a good start but doesn't cover all aspects of data quality. Monitoring distribution shifts, unique value ratios, and data type counts in production can help detect issues that result in "weird data." Tools like whylogs can be used to set up data quality monitoring on Kafka streams, offering lightweight statistical representations of data called profiles. These profiles can be compared, visualized, and monitored for changes, helping identify potential data quality issues early on.

WhyLabs has released a new version of its monitoring system for machine learning models and data streams. The updated system allows users to create custom monitors tailored to their specific needs, as well as switch on preset monitors with zero configuration. This enables efficient monitoring of models with thousands of features in just a few clicks. Additionally, the system minimizes alert fatigue by maximizing the helpfulness of alerts. The new version also includes enhanced customization options for severity and notification patterns to achieve reliable alerting.

The article discusses the challenges of monitoring image data within machine learning ecosystems. It highlights that maintaining observability is crucial as data volumes grow and complexities increase. The article suggests that monitoring unstructured data such as images can be achieved by capturing structured telemetry, which is compatible with common statistical approaches. It also mentions various physical factors like device settings, changes in environment, and object detection that can impact the consistency and quality of image data. Furthermore, it discusses data pipeline factors like swapped color channels, inconsistent color spaces, and scaling issues that can introduce points of failure. The article proposes a solution by computing metrics sensitive to these events such as mean pixel value for brightness, hue and saturation for color palette, and image height and channel count for colorspace. It also mentions the use of Exif data for additional information like geolocation. Finally, it introduces whylogs, an open-source data logging library designed to capture valuable telemetry in a customizable way for any dataset, which can be used with powerful anomaly detection, informative visualizations, and automated notifications through the WhyLabs AI Observatory.