Rust: Investigating a Strange Out-of-Memory Error

At Qovery, a small service called engine-gateway, responsible for authentication/authorization and data streaming, experienced unexpected restarts due to out-of-memory (OOM) errors despite stable memory usage monitoring. The issue, which initially lacked clear indicators, was traced back to a subtle coding oversight involving the Rust library `anyhow` and its error logging. The memory surge was caused by the library capturing and symbolizing backtraces when errors were logged using the Debug mode, which was not apparent in the monitoring due to its rapid occurrence within a 10-second interval. After increasing the memory limit and integrating the `jemalloc` allocator for detailed profiling, the team identified that changing a single line to avoid using Debug mode during error logging resolved the problem, highlighting the importance of understanding documentation and the potential pitfalls of default library behaviors.