AI & Cybersecurity: Learnings from three months of Semgrep Assistant

Semgrep Assistant, initially launched in a private beta three months ago, is now available to all Semgrep users on GitHub, aiming to enhance cybersecurity efficiency by leveraging AI to assist in static code analysis. The tool reviews security alerts and assesses whether findings are false positives or require code updates, integrating its recommendations into GitHub pull request comments and Slack notifications. During the beta phase, Semgrep Assistant marked 230 findings as likely false positives, with users agreeing with 95% of these assessments, although response rates were low. Data from the largest beta customer showed users were 1.5 times more likely to fix true positives and 2.2 times more likely to ignore false positives, indicating improvement over time, partly due to enhanced prompt engineering and GPT-4's updates. The initiative faces challenges such as extracting confidence levels from GPT-4's autofix suggestions and incorporating AI-driven rule writing, given the model's training data limitations. Future directions include refining auto-triage capabilities, better rule classification, and feedback loops for improved data quality, with community input sought for further development.