How we built an AppSec AI that security researchers agree with 96% of the time

Semgrep Assistant, by 2025, achieves a 96% alignment with security researchers in identifying true positive security findings, making it a reliable tool for filtering non-exploitable issues and alerting developers only when real problems exist. This high accuracy was reached through continuous benchmarking and refinement of the Assistant's decision-making processes using large language models (LLMs), which excel at reading and interpreting code. Despite being conservative in identifying false positives, leading to a 41% agreement rate in this area, the Assistant rarely overlooks true positives, ensuring critical issues are not ignored. The system effectively combines AI triage with human review, increasing developer efficiency by focusing attention on actionable findings and reducing noise, ultimately enhancing the overall security posture by encouraging developers to address more issues promptly. This AI-driven approach offers a significant advantage, especially for AppSec teams dealing with extensive backlogs, as it saves time and resources by automating parts of the triage process, though it does not replace the need for human expertise entirely.