May 2022 Summaries

The Named Entity Recognition (NER) task is an important component of various Natural Language Processing (NLP) pipelines, particularly challenging to improve due to the nuance and complexity of annotating text data. NER involves identifying words or spans in a sample that belong to specific label categories, such as person or location. Unlike text classification, where each sentence is classified into one category, NER can have multiple labels for each span, creating an explosion of potential tasks. However, collecting high-quality NER data is time-consuming and often requires domain experts, leading to limitations in training models. As a result, NER systems are often combined with rule-based features or fine-tuned on custom data using pre-trained language models. The black-box nature of NER models makes introspection and generalization efforts difficult, but Galileo's data-centric approach aims to address these challenges by surfacing error patterns and providing granular insights. By reducing the complexity of data structure and tagging schemas, Galileo enables more efficient model iterations and improves the performance of NER systems.

Galileo tackles data quality issues by analyzing various benchmark datasets in academia/industry using its platform, highlighting crucial errors and ambiguities within minutes. By inspecting a dataset like the 20 Newsgroups classification task, Galileo identifies 6.5% of malformed samples across the dataset, including empty or ill-formed samples that increase confusion during training. Using Galileo's Data Error Potential (DEP) Score, the platform quickly uncovers data errors that are otherwise found through ad-hoc exploration, enabling rapid discovery and fixing of these issues. By addressing these dataset errors, model performance improves, with a 7.24% overall performance improvement in this experiment, highlighting the importance of ML Data Intelligence to solve for necessary steps in the ML lifecycle.

The ML data problem is a significant challenge in building and deploying machine learning models, with errors and biases creeping into datasets and causing catastrophic repercussions. Data curation is complex and often overlooked, leading to model blindspots. Labels are also prone to error, and reused datasets can lead to data staleness. The lack of tools to address these issues has hindered the development of machine learning in enterprises, but a new solution called Galileo aims to provide insights and answers necessary to rapidly identify and fix data errors.