September 2024 Summaries

Semgrep has introduced enhanced reporting features to provide Application Security (AppSec) teams with greater clarity on production backlogs, developer engagement, and overall security posture. These updates include insights into the adoption of secure guardrails, which are designed to subtly guide developers towards secure coding practices without halting progress. The new reporting capabilities allow teams to track the effectiveness of secure guardrails by monitoring how many vulnerabilities are prevented in the developer workflow and how many findings are addressed before reaching production. Additional views include analyses of production backlogs, backlog activity, the most vulnerable projects, and the median age of open findings, offering a comprehensive understanding of security metrics. These tools aim to help AppSec teams validate the success of their "shift left" security initiatives and demonstrate positive trends in security management.

Selecting the right Application Security (AppSec) tool involves a comprehensive framework to ensure it aligns with an organization's specific needs and operational environment. Key steps include gathering feedback from industry peers and online reviews, deciding between first or second-generation static analysis tools based on security precision versus speed, and ensuring compatibility with the team's programming languages and frameworks. It's crucial to consider the tool's integration capabilities with existing systems, the management overhead if server maintenance is required, and the willingness to share source code with vendors. Financial considerations and the tool's customizability are also important, as is the ability to create bespoke security rules. Before making a final decision, conducting a proof of concept with developers is essential to ensure the tool meets operational expectations and gains developer buy-in for successful implementation, ultimately leading to more secure code with minimal friction.

This article concludes a series on building security champions by offering practical advice for implementing and sustaining such a program. Key strategies include defining program goals, managing time expectations, and engaging potential champions who might not be on the security team but show interest through courses. The article emphasizes the importance of a supportive security team culture and suggests recording group sessions to create onboarding videos for new champions or IT staff. A 'train the trainer' package is proposed to empower champions to educate their teams, complete with slides, demo instructions, and explanatory videos. The author encourages persistence, highlighting that the program's value will grow over time and inviting readers to share their success stories.

The integration of the Exploit Prediction Scoring System (EPSS) into Semgrep Supply Chain provides a powerful tool for prioritizing the remediation of security vulnerabilities by assigning dynamic scores that are updated daily, reflecting the likelihood of exploitation within the next 30 days. Developed by the Forum of Incident Response and Security Teams (FIRST), EPSS uses a machine learning model that draws on real-world data to help security teams focus on vulnerabilities that pose the most immediate risk. Although not customized for individual codebases, EPSS serves as a critical prioritization tool alongside Semgrep's dataflow reachability analysis, which reduces false positives by over 90% and enables teams to address the most actionable findings. While EPSS effectively predicts exploitation likelihood, it does not guarantee future exploits or assess vulnerability severity, thus it should be used in conjunction with other tools like CVSS for a comprehensive security strategy. By leveraging EPSS and Semgrep's capabilities, security teams can strategically allocate resources, focusing on the most pressing vulnerabilities, and ensure remediation efforts are impactful.

The text provides insights into the experiences and lessons learned by the AI team at Semgrep while implementing and evaluating Large Language Model (LLM) features. The team categorizes quality metrics into Behavior, Feedback, and Laboratory, each with its own challenges and benefits. Behavior metrics focus on the intended impact of AI features, while Feedback metrics involve user-generated ratings that can be biased or require segmentation for accuracy. Laboratory metrics, which include reproducible test suites and team evaluations, offer a shorter feedback loop but demand significant infrastructure. The document discusses the complexities of setting up a testing system that mirrors production environments, emphasizing the need for immutable and serializable template variables and the avoidance of dynamic rendering. It introduces the promptfoo tool, which facilitates LLM testing by managing prompts, template variables, and providers, and highlights its benefits over proprietary systems. The team employs various strategies to gather template variables, including using a staging database for simpler features and capturing real-world data for more complex scenarios. The system allows for the rapid evaluation of model variations and prompt adjustments, aiding in confident decision-making for model upgrades and feature enhancements.

Metrics play a crucial role in both reporting success to management and improving personal and program performance, as they provide evidence of effectiveness and areas for enhancement. However, it's important to distinguish between meaningful metrics and vanity metrics, which may look impressive but lack real value. An example highlights the difference between mere clicks and engagement, demonstrating that time spent reading an article reveals more about its impact than just the number of clicks. With this insight, strategic adjustments to platform focus can enhance engagement. In the context of a security champion program, relevant metrics include the number of new champions, engagement levels, bug reporting and fixing, and instances where champions identify previously unknown security issues. These metrics help demonstrate the program's ROI and effectiveness, while also gathering stories of success that can inspire and inform upper management.

Semgrep Code has enhanced its static application security testing (SAST) capabilities by incorporating framework-specific analysis, particularly for popular Python frameworks like Django, Flask, and FastAPI. Traditional SAST tools often struggle with frameworks due to the implicit control and data flows they introduce, which can obscure potential security vulnerabilities. By understanding the unique execution patterns and data handling mechanisms of these frameworks, Semgrep Code can effectively trace both control and data flows, capturing implicit paths and global object interactions that are critical for identifying security issues. The tool's framework-specific rules and analysis capabilities lead to a reported 84% true positive rate in benchmark tests, offering comprehensive coverage for security vulnerabilities in Python applications. This approach allows Semgrep to provide precise and efficient security scanning, making it a valuable tool for development teams using Python frameworks.

Semgrep Assistant has significantly enhanced its remediation guidance, achieving over 95% coverage of true positive findings and reducing the median time-to-resolution by 15%, saving developers an average of 20 minutes per finding. This improvement stems from providing step-by-step instructions and actionable human language explanations, which are particularly beneficial for junior developers or those lacking security expertise. The guidance, which is now integrated into developers' workflows such as PR comments and Jira tickets, not only helps in resolving current security issues but also educates developers, preventing future mistakes. For AppSec engineers, the detailed guidance simplifies scanning results and allows them to focus on critical issues without writing personalized code reviews, effectively scaling their impact. The AI-generated guidance is routinely evaluated by a team of security researchers to maintain high quality, and it has been found to be actionable nearly 78% of the time. These enhancements have led to a higher fix rate and faster resolution times, reflecting the value of AI in assisting developers with secure coding practices.