March 2023 Summaries

CleanVision is an open-source Python library that scans image datasets for common real-world issues such as blurry, under/over-exposed, oddly sized, or (near) duplicates of others. This can significantly impact the performance of machine learning models trained on these datasets. Issues detected in popular computer vision datasets like Caltech-256, Food101, CUB-200-2011, and CIFAR-10 include grayscale images, low information content, blurry images, near duplicates, odd aspect ratios, and mislabeled images. These issues can hinder the training of the best possible model on your data and lead to noisy or spurious correlations in the model's outputs. CleanVision offers a systematic approach for detecting these issues using just a few lines of code and can be used to audit most image datasets on a CPU. The library has been successfully tested on multiple famous image datasets, including CIFAR-10, which had the least number of issues amongst the ones evaluated here. By filtering out bad data, CleanVision helps improve the quality of datasets for various computer vision tasks.

KerasWrapperModel` is a one-line wrapper that enables TensorFlow/Keras models to be used with scikit-learn's rich ecosystem, including features like Pipeline and GridSearch. This allows users to leverage the strengths of both frameworks without having to rewrite their code or compromise on model architecture. The `CleanLearning` utility can also be applied to any sklearn-compatible model to identify label issues in the dataset and train a more robust version of the same model. By making neural networks sklearn-compatible, developers can tap into the full range of scikit-learn's functionality, including hyperparameter tuning and data preprocessing, to improve their models' performance and accuracy.

ActiveLab is an open-sourced active learning method that helps determine which new data should be labeled or which current labels should be checked again to improve machine learning models within a limited annotation budget. It uses a weighted ensemble of model-based predictions and annotator reliability estimates to decide on the most informative next label. By accounting for multiple annotators, agreement, and model confidence, ActiveLab outperforms other active learning methods in various settings, including multi-annotator and single-annotator scenarios with or without infinite unlabeled data. The method can be used to improve dataset labels, train better classifiers, and estimate annotator quality. It is available as part of the cleanlab library and has a tutorial notebook that provides a simple way to use it.

The cleanlab library has evolved from its original purpose as a proof-of-concept for utilizing ML models to discover mislabeled data, to become an industry-grade library that handles label errors in various ML tasks such as entity recognition, image/document tagging, and data labeled by multiple annotators. The latest release, cleanlab 2.3, introduces several new features including active learning with ActiveLab, which automatically answers the question of which new data should be labeled or which existing labels should be checked again, KerasWrapper for TensorFlow/Keras models, and improved computational efficiency for detecting label issues. These advancements aim to provide functionalities needed to practice data-centric AI, enabling users to improve their data and train better ML models with minimal labeling effort.