December 2022 Summaries

The goal of any machine learning (ML) project is to produce high-quality models quickly, but in reality, each ML project takes months from identifying the problem and use case to deploying the model in production. High-quality data is crucial for building high-quality models, as it's the most significant impediment to seamless ML adoption across the enterprise. To build a platform that helps curate high-quality models through high-quality datasets, it's essential to understand how your data is distributed, including its semantic coverage, outliers, noise, and semantically confusing features. A good machine learning platform should be able to evaluate a model on a hybrid set of metrics, including prediction latency, feature importance, and class balance. The system should also enable users to monitor and observe custom combinations of key metrics and provide actionable steps to fix issues automatically. By focusing on quality over quantity and using techniques like active learning and pre-trained embeddings, developers can build high-quality models that meet the real-world demands of their business.

The main bottlenecks in putting a high-quality Machine Learning (ML) model into production are data quality issues and inefficient workflows. Most ML teams spend 80% of their time fixing and improving data sets to get better model performance, rather than focusing on experimentation or deployment. This leads to problems such as mispredictions, bias, and slow deployment cycles. To solve these issues, Galileo is developing an "ML data intelligence" platform that automates the analysis and iteration process for ML teams, providing critical insights on how to improve model accuracy and efficiency. The platform works by hooking into existing model training frameworks and providing features such as dataset ranking, annotation error detection, and drift information on unstructured data. By incorporating these signals into an efficient automation workflow, Galileo aims to help ML teams ship models faster and get high-quality models faster.

A survey on machine learning data quality aims to address challenges related to data quality issues, such as the time spent finding and correcting errors in training data. Participants are encouraged to share information about the types of data they work with, the errors they encounter, and the tools they use. The anonymized survey responses will help guide discussions and solutions within the ML community, contribute to a free report with insights from experienced ML practitioners, and inform the development of a free instant ML data debugging tool. Additionally, a data quality event will be organized to discuss the survey findings with ML experts and data quality partners. Participants will receive an early copy of the report and have a chance to win Amazon gift cards as a token of appreciation.

Over 90% of the world's information is now unstructured, growing at a rate of 55-65% per year. This data is crucial for building AI-powered applications, but many challenges arise from its quality rather than algorithmic sophistication. Effective data scientists rely on intuition about their domain and data nuances to understand dataset curation strategies. However, curating datasets often involves more than just sampling data uniformly across labels, as it requires identifying necessary features to cover the variation in unstructured data. Domain expertise is also essential for understanding model behavior, annotating data, and fixing long-tail errors. With deep learning algorithms becoming commoditized, businesses are shifting focus from algorithmic improvements to dataset quality and using tools like Galileo to proactively find and fix errors in their datasets, ensuring models perform optimally in production.

At Galileo, the team aimed to enable machine learning engineers to surface critical issues in their datasets efficiently and quickly. They faced a unique challenge due to the large size of their data, which was not addressed by other model-centric ML tools. To address this, they explored various options, including using out-of-core data handling tools like pandas with chunking and pyarrow. However, these approaches proved insufficient or too slow for their needs. The team then turned to Vaex, a little-known project that offers an out-of-core hybrid Apache Arrow/NumPy DataFrame solution for big tabular data. Vaex provided the necessary scalability, performance, and flexibility to handle massive datasets with low latency. By leveraging its memory-mappable file format, chunking, and caching capabilities, the team was able to efficiently process large amounts of data, including single columns of high-dimensional vectors, and apply custom NumPy expressions using the `register_function` decorator. Vaex's performance and scalability were significantly better than other tools in handling high-dimensional data, making it an ideal choice for their platform.