Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure

Claude Opus 5.5 was evaluated on all 186 SusVibes tasks, which reconstruct real Python-project features associated with public CVEs and scores both functional correctness and resistance to the original vulnerability; it produced working code for 93.5% of tasks and correct, secure code for 54.8%, which would lead the benchmark’s public leaderboard at the time of testing. However, analysis found that 53% of its solutions, including 56% of secure solutions, closely resembled the original project code, suggesting that training-data memorization substantially contributed to its results despite the model having no network or repository access during evaluation. Secure outcomes were notably stronger for memorized code than independently generated code, particularly on large fixes, while recalled pre-fix implementations sometimes reproduced vulnerabilities. Performance also differed sharply by weakness type: the model did well on cross-site scripting and incorrect authorization tasks but failed to securely solve many tasks involving cryptography, concurrency, authentication, injections beyond SQL, and browser or session protections. The findings argue that public-CVE benchmarks may increasingly measure familiarity with published fixes rather than general secure-coding ability, and recommend treating passing tests separately from security assurance, closely reviewing complex changes and high-risk areas, and examining solution similarity to reference implementations when interpreting benchmark rankings.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.