Beyond working set memory: understanding the cAdvisor memory metrics

An investigation into cAdvisor memory metrics at Sourcegraph revealed misunderstandings regarding how memory usage is tracked and optimized for the Zoekt search engine powering Sourcegraph's code search. An engineer at Sourcegraph initially focused on resolving memory leaks and optimizing Go application resource usage, but found that key memory metrics, such as container_memory_working_set_bytes, did not behave as expected. This metric, previously thought to closely correlate with Go heap and stack usage, was found to be a loose estimate, influenced by filesystem cache usage rather than directly reflecting memory actively utilized by applications. Consequently, Sourcegraph's team refactored their memory dashboards to track additional metrics like container_memory_rss and container_memory_total_active_file_bytes, providing better insights into memory usage patterns and informing decisions on resource allocation. This comprehensive approach also highlighted the impact of memory mapping (mmap) on search latency, emphasizing the need for a well-sized filesystem cache to optimize search performance.