LLMs can structure data as well as humans, but 100x faster

The text discusses the development and evaluation of a benchmark for assessing the performance of large language models (LLMs) in labeling text datasets, comparing their effectiveness to human annotators. The study finds that state-of-the-art LLMs, such as GPT-4, can label text with equal or better quality than human annotators while being significantly faster and cheaper. GPT-4 achieves an 88.4% agreement with ground truth labels, outperforming human annotators' 86% agreement, and offers a favorable trade-off between label quality and cost. The report also highlights the use of confidence estimation to mitigate hallucinations and improve label quality, suggesting that combining different LLMs for various tasks can optimize performance. Additionally, the text emphasizes the potential of in-context learning and chain-of-thought prompting for enhancing LLM label quality and discusses ongoing efforts to expand the benchmark with more datasets, tasks, and models. The findings are facilitated by the Autolabel library, which has been open-sourced to encourage community collaboration and improvement.