April 2023 Summaries

The Office-Home Dataset is a widely used computer vision dataset that contains hundreds of erroneous labels and data issues, including mislabeled examples, ambiguous examples, and outliers, which can be detrimental to modeling and analytics efforts. These errors were discovered using Cleanlab Studio, an automated solution that identifies and fixes data issues using AI. The dataset was curated by collecting images from a web crawler and filtering them to ensure the desired object was in the picture, but this method often produces incorrect image-label pairs. By running the dataset through Cleanlab Studio, researchers can identify and correct these errors, which can improve the accuracy of their models and conclusions.

Cleanlab Studio` is an AI platform used to detect and fix issues in data, including human feedback provided during the training of Large Language Models (LLMs) like `Anthropic's Claude`. The dataset `hh-rlhf` from `Hugging Face Datasets` was analyzed using Cleanlab Studio, revealing various problems with the data. Examples include rejected outputs being better than chosen outputs due to human mistakes, and chosen outputs merely describing a subject without answering a query. These issues can hinder the reliability of LLMs trained via Reinforcement Learning from Human Feedback (RLHF). By running datasets through Cleanlab Studio, organizations can identify and fix such problems, leading to more reliable Large Language models.