The case of the vanishing CPU: A Linux kernel debugging story

A mysterious CPU spike in ClickHouse Cloud on GCP led to months of debugging, revealing a deeper issue within the Linux kernel's memory management. The investigation began with an occasional hiccup in ClickHouse Cloud infrastructure that engineers struggled to explain. Initially, it seemed like a random performance degradation issue, but further analysis revealed a hidden livelock caused by excessive contention on the `mmap_lock` spinlock. The lock was held for an exceptionally long time during page fault handling, scanning 1,093,267 pages in an effort to reclaim memory for the cgroup. This led to a situation where nearly all pages were activated, but only 32 pages were successfully reclaimed. A new bug was later discovered, which involved a spinlock called `lru_lock` protecting the struct `lruvec`. The fix for this issue involved enabling the Multi-Gen LRU (MGLRU) mechanism, which is designed to improve memory management and reduce contention on the spinlock. This change resolved the issue and improved overall system performance.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.