OOMProf - Profiling on the Brink

The blog post describes the challenges and solutions related to debugging programs terminated by the Linux Out-Of-Memory (OOM) killer, a kernel feature that terminates processes when the system runs out of memory. It highlights the difficulties developers face in diagnosing OOM events due to the lack of immediate indicators of the root cause, often complicated by memory overcommitment in Linux. To address these challenges, the authors developed OOMProf, a monitoring tool built on eBPF that profiles Go programs at the moment they are killed by the OOM killer. The tool involves a two-step process using tracepoints to identify and profile targeted processes and is integrated into the Parca Agent for continuous monitoring. While it primarily targets Go applications, its future plans include expanding support to other memory allocators and languages, enhancing the tool's capabilities in capturing comprehensive program states under OOM conditions. The post is interspersed with lyrics from a modern folk murder ballad, adding a creative narrative layer to the technical discussion.