CWE-bench v2: More coverage, quality, and transparency
Blog post from Hugging Face
Collinear AI’s CWE-bench v2 is an open benchmark of 100 cybersecurity repair tasks designed to evaluate whether AI agents can identify and fix real security weaknesses in open-source repositories while preserving existing functionality. The tasks span 91 Common Weakness Enumerations, use deterministic programmatic grading, undergo hand review and LLM-assisted verifier curation, and include safeguards against solutions that merely reproduce known upstream patches. A separate held-out task set remains available for private evaluation, while the public dataset, grading harness, and leaderboards support reproducible community comparisons across open-weight and API-only models. Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, narrowly surpassing Claude Opus 5.5, although the top model still fails 19% of tasks per attempt; DeepSeek V4.1 Flash, Claude Opus 5.5, and Gemini 4 Argon are highlighted as a cost-performance frontier. The benchmark can be run through the Harbor framework from Hugging Face, and its Dockerized environments and verifiers may also be used to train defensive agents.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| Gemini 4 Argon | 2 | 113 | 29 | 19 | -67% |
| Opus 5.5 | 2 | 1,032 | 79 | 41 | +65% |
| LLM | 1 | 3,355 | 742 | 191 | -49% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.