Company
Date Published
Author
Henrik Nordvik
Word count
789
Language
-
Hacker News points
None

Summary

In May 2017, Henrik Nordvik detailed efforts to resolve a native memory leak in Elasticsearch version 5.1.1, particularly affecting small clusters with minimal activity. The issue was identified when memory usage by Elasticsearch continually increased until the system rebooted due to memory exhaustion, indicating a leak. Although traditional tools like Yourkit and MAT didn't reveal the problem within the heap, enabling Native Memory Tracking (NMT) pinpointed the leak outside the JVM's tracking capabilities. To address this, jemalloc, a memory allocation library, was used to identify native memory allocations, revealing that java.util.zip.Inflater and java.util.zip.Deflater were major contributors due to improper memory release. These findings prompted fixes in both Lucene and Elasticsearch, resolving the issue in version 5.2.1. The investigation highlighted the importance of using advanced memory profiling techniques and tools beyond traditional JVM methods to detect and address native memory leaks in complex systems.