Secrets Detection – Tools for reproducible, detailed, and meaningful benchmarks

Samuel Guillaume discusses the process and strategies implemented at GitGuardian to enhance the performance of their secrets detection engine. The focus is on achieving a balance between speed, precision, and recall, given the vast amounts of data needing swift analysis. Key improvements include making the engine three times faster while expanding the number of detectors by 25% within a year. This was accomplished through rigorous benchmarking, which involves using a representative dataset of GitHub commits and a dedicated server to ensure reliable performance comparisons. The benchmarks, conducted at various stages of development, help identify performance bottlenecks, especially among detectors. Significant speed gains were noted with improvements to pre-validators, contributing to a nearly twofold increase in overall engine speed. The article emphasizes the importance of continuous performance monitoring and strategic enhancements, setting the stage for further detailed discussions in upcoming articles.