December 2025 Summaries

Language models (LLMs) have the capability to convert unstructured text into structured, business-ready data, but they are prone to errors that necessitate human review. The article explores methods to evaluate the trustworthiness of structured outputs from LLMs, focusing on Cleanlab's Trustworthy Language Model (TLM) as a solution that scores the reliability of LLM outputs and individual fields. By employing benchmarks across various datasets and models, Cleanlab’s trust scores proved to be 25% more precise and accurate in detecting errors compared to traditional methods such as LLM-as-a-judge and token log probabilities. TLM provides per-field trust scores, offering targeted oversight and allowing human reviewers to focus on the 1-5% of cases where LLM outputs are unreliable. This approach is especially efficient as it reduces the need for excessive model calls per field, making it scalable for complex outputs. The study concludes that traditional scoring methods are suboptimal for nuanced, field-specific accuracy, and emphasizes the value of Cleanlab's real-time scoring as an essential tool for enhancing the reliability of LLM-automated processes.

In a recent analysis of Structured Outputs from leading Large Language Models (LLMs), significant errors were found in the ground-truth outputs of popular benchmark datasets, highlighting the challenges in producing accurate annotations even by human annotators. To address this, four new Structured Outputs benchmarks with verified high-quality ground-truth outputs have been introduced, covering diverse applications such as Data Table Analysis, Insurance Claims Extraction, Financial Entities Extraction, and PII Extraction. These benchmarks are designed to facilitate more reliable evaluation of LLMs’ capabilities by providing clear and consistent annotations. The study compared the performance of various LLMs, including OpenAI's GPT-5 and GPT-4.1-mini, and Google's Gemini models, using metrics like Field Accuracy and Output Accuracy. Despite its higher cost, GPT-5 demonstrated superior performance in Financial Entities Extraction, while Gemini-3-Pro excelled in Data Table Analysis. The analysis suggests that while smaller models like GPT-4.1-mini and Gemini-2.5-Flash can offer cost and latency advantages, OpenAI's models are generally recommended for Structured Output tasks due to their cost-effectiveness, though accuracy may vary depending on the specific use case.

AI agents in customer service often struggle with multi-turn, tool-use tasks due to erroneous outputs from language models, which can derail interactions. One effective solution to mitigate these errors is real-time trust scoring, which assesses the reliability of each language model output. This method can significantly reduce agent failure rates, as demonstrated by the Tau²-Bench benchmark across domains like airline, retail, and telecom. When outputs are deemed untrustworthy, two strategies are considered: escalating the interaction to a human support representative or autonomously revising the message. The Cleanlab's Trustworthy Language Model (TLM) provides precise trust scores for detecting errors such as reasoning mistakes or incorrect tool calls. Automated escalation and message revision pipelines have shown to effectively decrease failure rates and improve success rates in customer interactions, as evidenced by tests using OpenAI's GPT-5 and GPT-4.1-mini models. These approaches offer a reliability layer for AI agents, enhancing their ability to handle complex tasks and reducing risks associated with incorrect outputs.