Investigating Linux Performance with Off-CPU Flame Graphs

The author, a performance engineer at SingleStore, encountered an uncommon Linux performance issue while running a synthetic workload on a columnstore table. The workload was using 16 threads to execute a simple query against the table, but the CPU cores were spending about 50% of their time idle. Using `perf_events` and an `awk` script, the author created an off-cpu flamegraph that revealed that every `mmap` syscall was taking around 10-20ms due to contention on the `mm->mmap_sem` lock. The issue was caused by SingleStore's use of `mmap`, which was inadvertently benchmarking Linux's `mmap` functionality, leading to significant performance overhead. To resolve the issue, the author switched from using `mmap` to the traditional file `read` interface, resulting in nearly doubled throughput and becoming CPU bound as expected.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.