JevBench explained: how decision models are benchmarked, and what it can't measure

JevBench is a name shared by several unrelated projects, with the best-known version operated independently by Benchmark Heaven as a leaderboard for typed decision models that answer from supplied state, instructions, rubrics, and labels. Its 1,500-item evaluation measures chance-adjusted accuracy, calibration, latency, and cost, combining these into Capability and Composite scores while using public and sealed questions, time limits, and cost and latency eligibility thresholds. In the November 2026 v1.8.7 results, baselabs mammoth-1 led the open-weights Capability ranking, Torchcast Decision 4B led that board’s Composite ranking, and Super Decider led hosted APIs by Composite; TypeSafe’s Jev models ranked lower, although Jev 1.13.0 serves as a reference for cost and latency caps. The benchmark is designed for tasks such as policy enforcement, claims triage, and tool-call guardrails, but cannot assess decisions requiring current external facts because all necessary information is included in each item. Parallel presents its PWS-Decisions benchmark as a complementary test of web-dependent questions, reporting that models with native web search were most accurate but slower and more expensive, while its own API emphasized lower latency and cost; these claims are accompanied by caveats about challenge-set selection, random-guess baselines, and non-web-grounded comparisons.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.