January 2025 Summaries

The authors of a Rust-based project were experiencing intermittent crashes on their ARM64 CI runners when running tests for an HTTP fetch feature. The crash was not reproducible locally, but was observed in the CI environment. After investigating, they discovered that the issue was caused by a race condition between threads accessing the same memory location while using `libc` functions like `getenv`. Specifically, the `setenv` function would move the environment block to a new location, causing the `getenv` function to access an invalid memory address when called concurrently. The authors were able to reproduce the issue by analyzing the assembly code and disassembling the `getenv` function. They eventually discovered that the problem was caused by the use of `rust-native-tls`'s `openssl-probe` which set the environment variables, but this was not thread-safe. To fix the issue, they decided to migrate away from `reqwest`'s `rust-native-tls` backend to `rustls` on Linux, or hold the Global Interpreter Lock (GIL) while calling `try_init_ssl_cert_env_vars`. The Rust project has already identified this as an issue and plans to make the environment-setter functions unsafe in the 2024 edition. Additionally, the glibc project has recently added more thread-safety to `getenv` by avoiding the `realloc` and leaking the older environments.

While porting network I/O code in EdgeDB from Python to Rust, developers encountered an unusual crash that only occurred on ARM64 CI runners during testing of a new HTTP fetch feature. Initially suspected to be a deadlock, investigations revealed it was caused by a crash in the libc `getenv` function, linked to a race condition when the `setenv` and `getenv` functions were called simultaneously in a multithreaded environment. The crash was traced back to the `openssl-probe` library setting environment variables, which conflicted with the native TLS backend used in the HTTP client library `reqwest`. The developers decided to migrate from the `rust-native-tls`/OpenSSL backend to `rustls` on Linux to circumvent the issue, acknowledging the complexities of handling environment variables in multithreaded applications. Future plans include addressing this with Rust's 2024 edition marking environment-setter functions as unsafe and recent glibc updates that improve `getenv`'s thread safety.