Content Deep Dive

CWE-bench v2: More coverage, quality, and transparency

Blog post from Hugging Face

Post Details
Company
Date Published
Author
Gonzalo Gonzalez, Ajay Jagannath, Nazneen Rajani, and Nathan Habib
Word Count
841
Company Posts That Month
60
Language
-
Hacker News Points
-
Post removed?
No
Summary

Collinear AI’s CWE-bench v2 is an open benchmark of 100 cybersecurity repair tasks designed to evaluate whether AI agents can identify and fix real security weaknesses in open-source repositories while preserving existing functionality. The tasks span 91 Common Weakness Enumerations, use deterministic programmatic grading, undergo hand review and LLM-assisted verifier curation, and include safeguards against solutions that merely reproduce known upstream patches. A separate held-out task set remains available for private evaluation, while the public dataset, grading harness, and leaderboards support reproducible community comparisons across open-weight and API-only models. Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, narrowly surpassing Claude Opus 5.5, although the top model still fails 19% of tasks per attempt; DeepSeek V4.1 Flash, Claude Opus 5.5, and Gemini 4 Argon are highlighted as a cost-performance frontier. The benchmark can be run through the Harbor framework from Hugging Face, and its Dockerized environments and verifiers may also be used to train defensive agents.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
Gemini 4 Argon 2 113 29 19 -67%
Opus 5.5 2 1,032 79 41 +65%
LLM 1 3,355 742 191 -49%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.