Jev vs. LLM-as-a-Judge: Accuracy and cost benchmarks

A benchmark of TypeSafe’s Jev, a typed probabilistic judging model, against Claude Opus 5 and GPT-5.6 Terra across 23,325 evaluations found that threshold tuning substantially affected hallucination-detection results. Using human-labeled RAGTruth and SummEval datasets, Jev matched Opus 5 at 87% hallucination-detection accuracy after validation-based threshold selection while costing about $0.05 per thousand judgments versus $14.30 and responding in roughly 141 milliseconds versus more than three seconds. Jev also achieved a 0.74 rank correlation with expert summary-quality ratings, near the estimated human agreement level and slightly above Opus 5 in this test. At a default 0.5 probability cutoff, however, Jev produced many false positives and appeared seven percentage points less accurate than Opus, despite similar ROC AUC scores, illustrating that probability outputs must be converted into decisions using thresholds suited to the application’s error tradeoffs. The evaluation recommends tuning thresholds on separate human-labeled validation data, checking results on held-out data, and retuning when tasks or models change; generative LLM judges may still be useful when written explanations are required.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.