December 2022 Summaries

The Robust & Responsible AI Newsletter - Issue #4 provides an overview of the latest developments in MLOps and Data-Centric AI. Key highlights include the announcement of the inaugural R2AI Summit, the release of The State of AI Report 2022, and new open source tools such as ZenML, MLflow 2.0, and Fugue with whylogs. Additionally, the newsletter features insights from industry experts on topics like NLP, technology readiness levels for ML systems, and responsible AI programs. The latest releases from WhyLabs include a guide to help businesses decide whether to build or buy an ML monitoring solution, integration highlights, and tips on how to monitor common issues with ML models in under 5 minutes. Upcoming events related to robust and responsible AI are also mentioned, along with opportunities for community engagement through the Rsqrd Community on Slack.

WhyLabs has introduced a Profile Store for real-time data monitoring on premises. The Profile Store allows users to write, list, and get profiles based on a time window or its dataset_id. It can be used as a reference to make validations and constraints against incoming data. An extension service for the Profile Store enables production use cases of whylogs on customers' premises with centralized and seamless integration. The Profile Store Service is a Docker-based REST application that can be deployed to their cloud infrastructure to better manage Profile Stores. It allows users to get, list, and write profiles to and from the Store and have it in sync with S3. In the future, its usage will be extended to keep profiles in sync with WhyLabs and other popular cloud storages.

This blog post discusses the use of statistical tests, specifically the Kolmogorov-Smirnov (KS) test, for detecting data drift in profiled datasets. Data drift is a common issue in machine learning applications that can degrade model performance if left unaddressed. The KS test is a nonparametric method used to compare two one-dimensional probability distributions and determine whether they are likely drawn from the same distribution. The post explores three experiments to better understand how different variables affect the results of the KS test when applied to data profiles: data volume, number of buckets, and profile size. The first experiment shows that increasing sample sizes increases the sensitivity of the KS test, making it more likely to detect small differences between distributions. However, this increased sensitivity may not always be desirable, as it can lead to false positives or overly conservative conclusions. The second experiment investigates how varying the number of buckets affects the accuracy of the KS test on data profiles. Results indicate that increasing the number of buckets generally reduces errors but also increases variance for larger sample sizes due to estimation errors in the profiling process. Finally, the third experiment examines the effect of profile size on error rates when using the KS test with data profiles. Results show that increasing the profile size decreases errors, making it possible to achieve results closer to the standard implementation by sacrificing some storage space. In conclusion, while performing the KS test on data profiles is feasible and produces results close to the standard implementation, there are limitations to consider. The sensitivity of the KS test increases with sample size, which may lead to false positives or overly conservative conclusions when testing under the null hypothesis. Additionally, tuning internal parameters such as profile size can improve accuracy but comes at a cost of increased storage requirements.