LLMScanBench Q3 2026: Choosing the Right Approach for AI Vulnerability Discovery | Harness Blog

A Harness benchmark compared GPT-5.5 and Claude Opus 4.8 LLM-based vulnerability scanners with Harness AI SAST across OWASP WebGoat, Juice Shop, and Benchmark for Java, evaluating precision, recall, consistency, speed, and cost using a shared Codex Security prompt workflow. The LLM scanners generally reported high precision, ranging from 85% to 100% on the benchmark applications, but recall was often low and highly variable, with both models missing many known vulnerabilities in WebGoat and Juice Shop; GPT-5.5 achieved 100% recall on Benchmark for Java while Opus 4.8 found very few issues there. The authors caution that the public, intentionally vulnerable applications may favor models trained on their source code, documentation, or embedded answer keys, while their ground-truth construction and LLM-assisted finding matching also shape the results. Repeated Opus scans of unchanged code produced materially different findings and recall rates, underscoring the probabilistic nature of LLM tools, whereas the deterministic AI SAST engine offered tunable precision-recall tradeoffs with faster and more predictable operation. LLM scans took roughly 1.6 to 37 times longer and cost 6 to 39 times more than AI SAST in the tests, though incremental GPT-5.5 scans reduced cost and time substantially for smaller code changes while potentially losing coverage on unusually large changes. Based on these findings, the authors recommend LLM-only scanning primarily for periodic deep assessments or penetration testing, while using deterministic scanners augmented by AI reasoning for continuous pull-request and pipeline security checks.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.