October 2022 Summaries

Researchers have developed two simpler methods for detecting Out-Of-Distribution (OOD) inputs, which are often neglected in complex OOD detection algorithms. These methods can be easily run on any type of data and do not require specific examples of outliers. The first method is based on feature embeddings, using the average distance to K-Nearest Neighbors (KNN) to score how atypical an example is. The second method uses predicted class probabilities output by a trained classifier to quantify uncertainty as a measure of outlyingness. Both methods are available as open-source code in cleanlab.outlier and can be used for various applications, such as autonomous vehicles where the system needs to detect unknown objects on the road.

This article presents a novel and simple adjustment to model predicted probabilities that can improve Out-of-Distribution (OOD) detection with classifier models trained on real-world data. The approach is based on theory and runs in just a couple of lines of code. It involves adjusting the model's predicted probabilities using class confident thresholds, which are calculated from the training data. This adjusted OOD detection procedure remains extremely simple and easy to implement in practical deployments. Experimental results show that this method improves the performance of both Entropy and MSP-based out-of-distribution detection scores.

The cleanlab package has been extended to entity recognition tasks, which involves annotating each word in a sentence with its corresponding label. The cleanlab package can now identify label errors in token classification data, including the CoNLL-2003 dataset, which is commonly used for benchmarking entity recognition models. The code provided by cleanlab allows users to easily find and fix issues in their datasets, using an open-source algorithm that has been proven to be effective in detecting label errors. The package also provides additional functions to help understand the dataset better, such as identifying the most commonly mislabeled words and determining what types of labels are most frequently assigned incorrectly. Overall, cleanlab is a useful tool for improving text data quality and developing reliable machine learning models.

A new open-source module called cleanlab.multiannotator has been developed for measuring the quality of multi-annotator classification data using novel CROWDLAB algorithms. The module can estimate consensus labels, quality scores for each consensus label and annotator, and is more effective than existing solutions on real-world data. It works by forming a probabilistic ensemble prediction considering the labels assigned by each annotator as outputs from other predictors. This approach allows CROWDLAB to still perform effectively even when the classifier is suboptimal or a few of the annotators often give incorrect labels.