May 2023 Summaries

The text discusses the issue of Independent and Identically Distributed (IID) data in machine learning, data science, and statistics/analytics efforts. It highlights that most datasets violate this assumption due to various reasons such as data drift, non-IID sampling, or lack of independence between datapoints. The authors present a method called k-Nearest Neighbors (kNN) to detect when a dataset is not IID. This method constructs a graph of the dataset based on feature values and applies a two-sample permutation test using the Kolmogorov-Smirnov statistic to determine if there is a statistically significant difference between the distributions of index distances between kNN-neighbors and arbitrary datapoint pairs. The text also showcases various examples, including image datasets with concept drift and extreme drift, where cleanlab's non-IID check successfully identifies the issues. Additionally, it highlights additional features such as scoring individual datapoints and handling identically distributed but not independent data.

To systematically improve any image, text, or tabular/CSV/Excel dataset, one can quickly run it through Cleanlab Studio — an automated solution to find and fix data issues using AI. The Stanford Cars dataset, originally used in a research paper with over 1000 citations, was analyzed for common issues and outliers, which were detected by Cleanlab Studio, revealing mislabeled images that affect product categorization and identification efforts in e-commerce analytics and business intelligence. These errors can have detrimental effects on modeling and analytics efforts, highlighting the importance of correcting them to produce accurate models and data-driven conclusions. Cleanlab Studio's universal Data-Centric AI platform can be used to find and fix issues in various datasets, including text, image, table/CSV/Excel, and more, offering a free solution for data improvement.

Active learning is a technique used in machine learning to select the most informative data points to annotate, reducing the need for human labeling and resulting in cost savings and faster model training times. In this article, ActiveLab is presented as an active learning algorithm that can be particularly useful when dealing with noisy annotators. The authors demonstrate the effectiveness of ActiveLab in improving the accuracy of a fine-tuned Hugging Face Transformer for text classification while keeping the total number of collected labels from human annotators low. By using ActiveLab, the authors achieve 90% model accuracy at only 35% of the label spend as standard training, outperforming random selection by a significant margin. The technique is particularly useful in text classification tasks, where annotating data can be time-consuming and expensive. Overall, ActiveLab offers a promising approach to improving datasets with minimal labeling effort.

Datalab is an open-source platform that automatically detects common real-world issues in datasets, such as label errors, outliers, near duplicates, non-IID sampling, and low-quality/ambiguous examples, without requiring manual domain knowledge. It utilizes any trained Machine Learning model to diagnose dataset problems that can be fixed to produce a better version of this model. Datalab operates on predictions and/or representations from any ML model already trained, allowing data scientists to quickly analyze their dataset for issues and improve the quality of their data before training a new model. By automatically flagging data issues, Datalab enables data scientists to build reliable models from unreliable datasets, and its open-sourced nature makes it easy to add custom data quality checks or contribute to its development.