Evaluating LLMs Under Production Parity: A Replay Pipeline for Safe Model Swapping in Conversational Agents

A production-parity replay pipeline was developed to evaluate conversational-agent model swaps by reconstructing validated synthetic sessions with the original prompts, skills, memory, tools, routing, and termination rules, while changing only the LLM making each decision. Rather than executing live actions, the system injects recorded tool results for matching calls and simulated failures for divergent ones, enabling safe and reproducible testing across metrics including tool accuracy, task completion, latency, cost proxies, behavioral quality, and hallucination rates. From an initial set of 106 approved sessions, manual review produced a 20-session reference corpus designed to ensure reliable baseline behavior. Eight models were tested, with GPT-5.4 mini, GPT-5.4 nano, and Kimi-K2.5 approved; Gemini 2.5 Flash, GPT-4.1 nano, and GPT-5 mini achieved relatively high composite scores but failed hallucination thresholds, while GPT-OSS-120B faced structural compatibility problems with its output format. The study’s central finding is that hallucination rates should function as independent, eliminatory safety gates rather than being diluted within weighted average scores, since strong performance in other categories can mask serious failures in customer-facing or critical workflows. Run-by-run analysis also showed that some aggregate rejections were statistically unstable, whereas GPT-OSS-120B’s repeated failures indicated a persistent structural issue, underscoring the need for repeated testing and ongoing monitoring for each model family.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.