April 2022 Summaries

In June 2021, Forbes published an article on the movement towards Data-Centric AI which revolves around the insight that improving the data rather than the model can be more effective in improving the overall performance of AI systems. This approach makes intuitive sense because the quality of Machine Learning (ML) models depends on the quality of the data used to train/evaluate them, as "garbage in, garbage out." Given the abundance of awesome open-source ML modeling packages, the model aspect is more-or-less a solved problem for many business applications, leaving a key challenge in making Data-Centric AI an efficient and systematic process. To address this, new tools focused on data quality for AI are needed. One such tool is cleanlab, which uses an algorithm called “Confident Learning” to automatically find label issues in any dataset. This tool has been used to uncover thousands of label errors in top 10 ML benchmark datasets, highlighting the importance of data quality even in well-studied datasets. The article demonstrates how to use cleanlab to find label issues in audio datasets used for supervised learning, using the Spoken Digit dataset as an example. By leveraging cleanlab, developers can identify and fix label errors in their own datasets, ensuring that their ML models are trained on high-quality data.

The text discusses the limitations of supervised machine learning and the importance of ensuring that training data is accurate. It highlights the issue of label errors in datasets, even those from real-world applications, which can lead to flawed models being deployed. To address this, the open-source cleanlab library provides a tool to identify label issues in datasets. The text then demonstrates how to use cleanlab on the MNIST dataset, which has been cited over 40,000 times, to find label issues and visualize specific examples that warrant closer inspection. The findings of cleanlab are used to demonstrate how to use the library to improve the accuracy of machine learning models by identifying and correcting label errors in datasets.

Cleanlab is an open-source framework for machine learning and analytics with messy, real-world data. It identifies errors in datasets, measures dataset quality, trains reliable models with noisy data, and helps curate high-quality datasets. The tool automates several workflows to help users practice more data-centric AI. Cleanlab significantly reduces the pain of data cleaning by automatically flagging only the small subset of data that truly requires attention. It supports workflows for machine learning and analytics with messy real-world data, finds and fixes example-level, class-level, and dataset-level issues; measures and tracks overall dataset quality; and provides cleaned data for machine learning pipelines.

Cleanlab is a startup that aims to make AI work reliably with noisy, real-world data and labels. Founded by Curtis Northcutt, the company spun out of his decade-long PhD research at MIT. The foundational work was done in collaboration with Isaac Chuang, inventor of the first working quantum computer. Cleanlab's mission is to empower AI to work with real-world data and solve human-centric problems that previously seemed unsolvable due to label errors. The company has grown rapidly since its incorporation in late 2021, with dozens of tech, healthcare, finance, and data-related companies using their technology. Cleanlab's approach is based on math and science published in top peer-reviewed conferences and journals in machine learning and AI. The company believes in free dissemination of ideas and information, which is why they make all their research contributions public and open-access, including algorithms, code, and datasets. In the future, Cleanlab envisions a world where AI can fulfill its promise to empower humanity, even though human data is messy. They aim to build a world where medical practitioners can trust machine learning models to make reliable diagnoses by using Cleanlab to train on cleaned data. The company's ultimate goal is to create a world where AI actually works for people and their real-world, messy data.