September 2025 Summaries

In the exploration of evaluation methods for large language models (LLMs), the study compares binary and score-based evaluations, revealing that while numeric scoring offers granular detail, it suffers from instability, with scores often collapsing into broad bands or plateaus, particularly in cases of spelling and structural errors. The 2025 tests, which utilized advanced models like GPT-5-nano and others, showed some improvements in consistency over previous years, but also confirmed persistent challenges in using numeric scales, as these often lack the reliability needed for nuanced judgments. Binary and multi-categorical rubrics, such as letter grades, provide more stable and reproducible results, aligning better with human annotations, though they sacrifice some sensitivity to finer distinctions. The research underscores a critical trade-off between stability and resolution in LLM evaluations, suggesting that while binary and categorical approaches are more consistent, numeric scores require tightly controlled conditions to be effective.

In a recent presentation, Arjun Mukerji, PhD, a Staff Data Scientist at Atropos Health, introduced RWESummary, a benchmark for evaluating large language models (LLMs) in the context of summarizing real-world evidence (RWE) studies. Mukerji emphasized the importance of selecting reliable AI models for healthcare due to its high-stakes nature, where errors can have significant consequences. RWESummary tests LLMs on converting structured study inputs into plain-English summaries, focusing on three key evaluations: the accuracy of the direction of effect, numerical accuracy, and completeness. Mukerji highlighted that getting the direction of effect right is crucial, as reversing it could lead to severe misinterpretations. The benchmark revealed no single model excelled in all areas; Gemini 2.5 performed best overall in accuracy, while Gemini 2.0 Flash was superior in speed. Mukerji advocated for robust evaluations and incorporating human oversight in AI-driven healthcare workflows to mitigate risks.

In an interview with Bobby Vinson, an AI Evaluation Engineer at Trunk Tools, the company discusses its innovative approach to revolutionizing the construction industry through AI agents, specifically designed to streamline data management, automate tasks, and reduce waste. Trunk Tools has developed a variety of AI agents, including a primary question-and-answer agent that assists construction teams by providing quick access to necessary information, thus enabling them to focus more on building rather than on administrative tasks. Vinson emphasizes the importance of not taking shortcuts in complex systems and highlights the critical role of evaluation in understanding the impact of changes within these systems. Observability is also crucial, as it helps in understanding the intricate workflows of the agents and in making informed decisions for future developments. The choice of the Arize AX platform was driven by its alignment with Trunk Tools' future needs and the commitment to a partnership that addresses emerging challenges in the rapidly evolving field of AI agent engineering.

Arize has benchmarked its database, adb, to assess its performance both as a standalone system and within its application, focusing on aspects such as dataset upload, trace ingestion, and full-text search. The benchmarks aimed to evaluate how quickly and efficiently adb can handle large data uploads and real-time data processing, including the speed at which large datasets can be ingested using a batch insertion method and how swiftly real-time events are made available in the user interface. Additionally, the tests assessed adb's capability to perform full-text searches across extensive data sets, specifically analyzing the speed of regular expression searches within large spans of chat text. The benchmarks demonstrated adb's effectiveness in handling substantial data volumes and maintaining fast processing times, with expectations for consistent performance under similar conditions.

The article delves into the intricacies of the orchestrator-worker agent architecture, a dynamic workflow model where an orchestrator agent decomposes tasks into subtasks handled by worker agents, iterating until a final output is achieved. This architecture diverges from prompt chaining by adapting to runtime conditions, with subtasks varying in complexity and requiring diverse strategies. Key challenges include dynamic routing, context continuity, seamless handoffs, error handling, concurrency, and memory management. The text compares leading frameworks—Agno, Autogen, CrewAI, OpenAI, LangGraph, and Mastra—by building orchestrator-worker prototypes to highlight how each implements this architecture, focusing on execution models, handoffs, memory, and error handling. Each framework offers distinct approaches to orchestration, from chat-driven methods like AutoGen to graph-based models like LangGraph, each with unique trade-offs and strengths, providing developers with options tailored to different needs. CrewAI and Mastra are noted for their robust guardrails and memory systems, Agno for its declarative coordination, OpenAI for seamless integration within its ecosystem, LangGraph for precise control, and AutoGen for its conversation-centric flexibility.

The article outlines a comprehensive framework for evaluating multilingual AI systems' ability to generate accurate Cypher queries from non-English inputs, crucial for global business applications. It presents a detailed evaluation pipeline using Phoenix for observability and DSPy for structured instrumentation, focusing on translating English questions into languages such as Hindi, Tamil, and Telugu, and assessing the quality of generated Cypher queries against English ground truth. The process includes translation quality assessment through back-translation and semantic similarity, cross-lingual Cypher accuracy evaluation using LLM judges, and rich metadata capture for detailed analysis. The framework highlights the challenges of maintaining technical accuracy in multilingual contexts and provides insights into translation quality's impact on downstream tasks, model robustness, and language-specific error patterns. The article emphasizes that translation quality alone does not ensure success in generating correct technical outputs and offers steps for expanding the evaluation to more languages and integrating it into production monitoring systems.

Stan Miasnikov, a Distinguished Engineer at Verizon, recently presented his paper on inter-agent communication, titled "Category-Theoretic Analysis of Inter-Agent Communication and Mutual Understanding Metric in Recursive Consciousness," during an AI research paper reading hosted by the Arize community. The paper explores the extension of a recursive consciousness framework to better understand the communication dynamics between agents, focusing on how meaning is often lost during translation. Miasnikov's work aims to provide deeper insights into how agents can achieve mutual understanding despite these challenges.

Peter Belcak, an AI researcher at NVIDIA, presents in his paper that small language models (SLMs) are the future of agentic AI due to their efficiency and cost-effectiveness in specific tasks within agentic systems. The paper argues that SLMs, defined as models with fewer than 10 billion parameters, can be as effective as larger language models (LLMs) in tasks like tool-calling, structured reasoning, and code orchestration, offering significant cost and efficiency benefits. Belcak suggests a pragmatic workflow where tasks are initially mapped with a large model, then specialized SLMs are implemented for specific jobs, fine-tuned iteratively for quality and efficiency. The paper emphasizes that heterogeneous systems, which utilize both SLMs and LLMs, are more suitable for achieving the best results in agentic AI, particularly in environments where resource optimization is crucial. The research also highlights the potential for SLMs to enable edge and on-device deployments, enhancing privacy and reducing latency for light-duty language components, while reserving more complex tasks for larger models.

Arize AX has introduced several updates aimed at enhancing the user experience and functionality for agent engineering, data visualization, and dataset management. The Experiments page now offers improved UX with richer charting capabilities, including enhanced diffing support and new select components for side-by-side analysis, alongside usability fixes such as expandable tables and editable experiment names. The tracing interface features a new dedicated Agent Graph tab for better visualization of agent interactions, and trace interactivity has been improved with hover states and clickable spans. In the Playground, performance updates ensure reliable data loading, and support for GPT-5 has been added for experimentation. Dataset management has been enhanced with improved filtering, expanded REST API capabilities, and support for images in datasets and labeling queues. Other updates include expanded annotation configuration capabilities, advancements in the Alyx Copilot API, and UX enhancements on the Projects page, along with a revamped evaluation and tasks experience. These updates collectively aim to streamline workflows and improve the overall user experience in Arize AX.

The AI Evals for Engineers & PMs course, led by Hamel Husain and Shreya Shankar, offers a comprehensive framework for evaluating and enhancing large language model (LLM) applications, as exemplified by the Recipe Bot Workflow. This hands-on course integrates open-source tools like Arize Phoenix and covers a systematic five-step evaluation process, including prompt design, synthetic data and error analysis, LLM-as-a-judge evaluators, retrieval evaluation for retrieval-augmented generation (RAG), and state-level diagnostics. Each step involves specific tasks such as designing and iterating prompts, using synthetic data to identify errors, employing LLMs for automated error judgment, and analyzing retrieval and pipeline states. Phoenix plays a crucial role by logging, tracing, and managing experiments, allowing participants to track progress and make data-driven improvements. This structured approach emphasizes reproducibility and scalability, moving from isolated debugging to a refined workflow that can adapt to increasing system complexity.