September 2023 Summaries

Cleanlab Object Detection is a novel algorithm that detects annotation errors and assesses the quality of labels in any object detection dataset. It has been open-sourced in the cleanlab library and can be applied to datasets like COCO 2017, where it automatically flags images with incorrect original labels. The algorithm utilizes existing trained object detection models to score the label quality of each image, allowing for prioritization of mislabeled images for review/correction. Cleanlab Object Detection outperforms other label quality scoring methods and is easy to use, requiring only a few lines of code after training any standard object detection model on the dataset.

Cleanlab Studio is a no-code platform that uses novel Data-Centric AI methods to improve the quality of image/text/tabular datasets and machine learning models, automatically detecting issues such as labeling errors, outliers, duplicates, and mislabeled examples in satellite imagery data. The RESISC45 dataset, with over 1,400 Google Scholar citations, is used as an example, where Cleanlab Studio detected 281 label issues, 363 outliers, and 20 near duplicates, revealing the potential for automated data correction to improve the reliability of ML models and analytics in critical situations like disaster response. By using Cleanlab Studio, users can automate the process of identifying and addressing these issues, leading to more accurate scientific research, policies, and financial outcomes, while also enabling the development of more complex and powerful machine learning models.

Our new algorithm detects errors in numerical datasets by fitting a regression model to predict values based on other columns in the dataset. It accounts for uncertainty sources and is more effective than conformal inference or RANSAC. The algorithm scores each datapoint based on how likely its value is correct, as well as estimating how many datapoints' values were corrupted. This allows for prioritizing the most-likely corrupted datapoints for subsequent review. Our approach works with any regression model that supports a standard fit() and predict() interface, making it versatile for various types of data including image, text, and audio data with associated numerical outcomes. The algorithm uses the regression model to estimate aleatoric and epistemic uncertainties, producing quality scores that reduce opportunities for incorrect datapoints to be misclassified. By sorting by these scores and fitting multiple copies of the regression model with bootstrap resampling, we can estimate the fraction of corrupted datapoints in the dataset. Our algorithm has been benchmarked on 5 real numerical datasets with naturally-occurring errors, showing significant improvements over alternative approaches like conformal inference and RANSAC. The cleanlab library provides a simple Python code to run this algorithm on your data, making it easily accessible for automatic validation of your own datasets.

The `cleanlab` package has been updated to version 2.5, adding support for regression tasks and improving its capabilities across various machine learning (ML) tasks. The new release includes features such as label error detection for object detection and image segmentation, automatic data quality improvement, and enhanced datalab functionality. Additionally, the package now supports `Cleanlab Studio`, a no-code platform that automates many steps in turning raw data into reliable ML models. With these updates, `cleanlab` has become one of the most comprehensive packages for data-centric AI, supporting multiple ML tasks and providing tools for improving data quality and model robustness.

The Cleanlab Studio Audit (CSA) is a tool that uses AI to identify issues in popular datasets, such as the Food-101N dataset, which contains 101k images with 101 food categories. The audit revealed thousands of label issues, outliers, ambiguous examples, and near-duplicates in this famous computer vision dataset. Cleanlab Studio automatically identified mislabeled images and suggested more appropriate labels, including a piece of cheesecake labeled as a carrot cake. The authors of the Food-101N dataset noted that it had more images and was noisier than the original Food-101 dataset, but did not mention these issues in their disclaimer. Cleanlab Studio found 27,488 mislabeled examples, 8,519 outliers, 13,538 ambiguous examples, and 17,510 near-duplicate examples, which are detrimental to modeling and analytics efforts. The tool helps data owners identify and fix such errors to train the best models and draw accurate conclusions.