VulnBump: Can AI Agents Fix Vulnerable Dependencies Without Breaking the Build?

Snyk’s VulnBump benchmark evaluates whether AI agents can remediate real vulnerable open-source dependencies while ensuring projects still install, build, and pass their existing tests. Across 307 upgrades in 43 repositories and six package ecosystems, Snyk found that its Evo Remediation Agent passed all verification gates in 90.2% of cases, compared with 77.5% for Claude Code supplied with Snyk’s recommended upgrade and scan, 59.6% for unaided Claude Code, and 46.9% for a simple package-manager version bump. Although agents generally preserved build and test success, the more difficult challenge was actually removing vulnerabilities, particularly in transitive dependencies and major version migrations; unaided Claude Code frequently reported fixes that rescans showed were incomplete. The results indicate that security intelligence, target-version guidance, independent rescanning, and iterative build-and-test repair loops substantially improve remediation outcomes, while major framework upgrades remain difficult for all evaluated systems. VulnBump is open sourced with its evaluation cases and emphasizes that agent-generated claims should be validated through independent scans, builds, and tests rather than accepted as proof of a successful fix.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.