June 2023 Summaries

OpenAI's newly released models, gpt-3.5-turbo-0613 and gpt-4-0613, were evaluated against their predecessors for their performance in labeling text datasets across various natural language processing (NLP) tasks. The gpt-3.5-turbo-0613 model demonstrated a 40% faster turnaround time than its predecessor but exhibited slightly lower label quality for six out of eight datasets. In contrast, the gpt-4-0613 model maintained a similar label quality to its predecessor while also being 20% faster. Despite these improvements in speed, the new models remain significantly faster than human annotators, with the gpt-4-0613 model outperforming human labeling quality. The evaluation suggests that while the newer models offer increased efficiency, developers are advised to assess performance on their specific data and use cases before transitioning to the updated models.

The text discusses the development and evaluation of a benchmark for assessing the performance of large language models (LLMs) in labeling text datasets, comparing their effectiveness to human annotators. The study finds that state-of-the-art LLMs, such as GPT-4, can label text with equal or better quality than human annotators while being significantly faster and cheaper. GPT-4 achieves an 88.4% agreement with ground truth labels, outperforming human annotators' 86% agreement, and offers a favorable trade-off between label quality and cost. The report also highlights the use of confidence estimation to mitigate hallucinations and improve label quality, suggesting that combining different LLMs for various tasks can optimize performance. Additionally, the text emphasizes the potential of in-context learning and chain-of-thought prompting for enhancing LLM label quality and discusses ongoing efforts to expand the benchmark with more datasets, tasks, and models. The findings are facilitated by the Autolabel library, which has been open-sourced to encourage community collaboration and improvement.