The best open models aren't on your leaderboard

TrustedRouter argues that widely cited AI leaderboards often lag behind current Chinese open-weight models and reports running models including DeepSeek, GLM, Kimi, MiniMax, MiMo, and Hunyuan on published factuality, reasoning, coding, instruction-following, long-context, and safety benchmarks through a common API. Its results suggest that leading open models can match or approach frontier proprietary systems on closed-book factuality and broad knowledge tests, while retaining substantial deficits in repository-level coding, where Claude Opus led by a wide margin. The report emphasizes benchmark validation after uncovering issues caused by token limits, provider transport failures, and an incorrectly reimplemented benchmark grader, arguing that reproducible harnesses and official scoring methods are essential. It also finds major variation in model safety refusals, with some models responding to nearly all unsafe prompts and others refusing all of them, and presents multi-model routing as a way to select models based on task performance, availability, cost, and desired safety policies rather than treating any single system as universally best.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.