Jev vs. LLM-as-a-Judge: Accuracy and cost benchmarks

A benchmark of TypeSafe’s Jev, a typed probabilistic judging model, against Claude Opus 5 and GPT-5.6 Terra found that threshold tuning substantially affects hallucination-detection results. Across 23,325 judgments on the human-labeled RAGTruth and SummEval datasets, Jev matched Opus 5’s 87% hallucination-detection accuracy after its probability cutoff was tuned on separate validation data, while costing about 300 times less and responding roughly 23 times faster. At the default 0.5 cutoff, however, Jev appeared seven percentage points less accurate because it falsely flagged many grounded responses, despite having nearly the same ROC AUC as Opus 5. On summary-quality ratings, Jev achieved a 0.74 rank correlation with expert assessments, comparable to or slightly above Opus 5 and near estimated human agreement, at far lower cost. The analysis argues that teams should calibrate any judge, including LLM-based judges, using several hundred human-labeled examples from their own traffic, choose thresholds based on acceptable false-positive and false-negative tradeoffs, and evaluate on held-out data. Jev’s low latency and cost may make it suitable for scoring every production response, while generative LLM judges remain useful when written explanations of failures are needed.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.