Lessons from building Sherlock: Automating security code reviews with Sourcegraph

Sherlock is an AI-powered tool developed by a security team to enhance code reviews by automating the identification of security risks while reducing false positives. Traditional SAST tools often overwhelm engineers with alerts, many of which are false, making it challenging to identify genuine vulnerabilities. Sherlock leverages Large Language Models (LLMs) and Sourcegraph Cody's contextual insights to provide richer analysis of pull requests and diffs. It integrates seamlessly with GitHub workflows, automatically correlating SAST alerts with LLM-driven insights to focus security efforts on high-risk issues. Despite challenges like occasional hallucinations and limited code navigation, the tool has proven effective, scanning over 400 pull requests and uncovering several high and medium-severity issues. Sherlock's ability to flag both straightforward vulnerabilities and nuanced edge cases enhances proactive code security reviews, saving time and allowing the security team to concentrate on more complex tasks. By prioritizing actual risks, Sherlock accelerates development cycles without compromising security, delivering tangible business benefits and boosting productivity.