How We Migrated the Heart of Our Platform to Rust

GitGuardian embarked on a challenging migration of their secret detection engine from Python to Rust, driven by the need for improved performance and portability. This transition aimed to enhance raw performance to reduce infrastructure costs and allow the engine to operate in environments without Python. The migration process involved rewriting the engine from scratch while maintaining backward compatibility, which posed significant challenges but avoided user disruption. The use of Rust brought a consistent performance boost across the engine as Python's abstraction costs were eliminated. However, the team faced issues with Python's Global Interpreter Lock (GIL) during long operations, which were resolved by releasing the GIL during Rust processes, leading to a notable speed increase. The migration also served as an extensive review of the legacy codebase, deepening the team's understanding and preparing them for future automation. Despite the complexities, the rewrite resulted in a threefold global speedup and set the stage for further improvements in secret detection efficiency.