June 2023 Summaries

The article highlights the importance of reliable model evaluation in MLops and LLMops, particularly in prompt selection for large language models (LLMs). It demonstrates that relying solely on observed test accuracy can lead to suboptimal choices due to noisy annotations. The study uses a binary classification variant of the Stanford Politeness Dataset and finds that the FLAN-T5 LLM performs better with certain prompts when assessed using cleaner test data, which more closely reflects actual model deployment performance. It emphasizes the need for high-quality evaluation data and suggests using software like Cleanlab to verify label quality before making critical decisions based on observed test accuracy.

This article demonstrates how Cleanlab Studio, an AI tool, can be used to automatically improve the accuracy of a legal judgement prediction model by identifying and correcting issues in large datasets. By using this tool, users can achieve top-quality data and improved models with just a few clicks. The article showcases the effectiveness of Cleanlab Studio on the European Court of Human Rights dataset, where it managed to obtain a 14% error reduction on model predictions compared to state-of-the-art methods like fine-tuned OpenAI Large Language Models.

The Fashion MNIST Dataset has been analyzed using Cleanlab Studio, an automated solution to find and fix data issues using AI. The audit revealed hundreds of erroneous labels and data issues in the dataset, which can affect product categorization and product identification efforts in e-commerce analytics and business intelligence. Mislabeling images, particularly those belonging to the t-shirt/top class, was a common issue, while footwear-type images were also found to be mislabeled. Additionally, Cleanlab Studio detected ambiguous examples and outliers that should be removed from the dataset entirely. The analysis highlights the importance of correcting data errors to train accurate models and draw reliable conclusions, and provides a tool for users to find and fix such issues in their own datasets.

This article demonstrates how data-centric AI tools like cleanlab can improve a fine-tuned Large Language Model (LLM) by optimizing the dataset itself, rather than altering the model architecture or hyperparameters. The authors use the Davinci model from OpenAI and achieve a 37% boost in test-set performance on a politeness classification task after removing data with automatically-flagged label issues and then fine-tuning the LLM on the filtered dataset. Similar gains are achieved for other state-of-the-art LLM models, Ada and Curie. The authors also introduce a no-code solution to efficiently fix label errors in the dataset using Cleanlab Studio, which reduces the error rate of the model by 37%. The article highlights the benefits of data-centric AI tools like cleanlab, which can help systematically engineer better data via automation, freeing up time for domain experts to focus on their unique knowledge.