A Netty ByteBuf Memory Leak Story and the Lessons Learned

In a detailed account of addressing memory leaks in a Netty-based log receiver at Logz.io, Asaf Mesika shares his experience of troubleshooting off-heap memory issues by utilizing Netty's advanced leak detection mode. Initially dismissing a single error message that highlighted the lack of a ByteBuf.release() call, Mesika later realized the importance of this alert as the host's free memory continued to decrease. By enabling advanced leak reporting through a JVM option, he was able to pinpoint the location of the leak in the code, leading to two key takeaways: promptly switching to advanced leak detection when suspecting memory leaks, and tracing the code hierarchy to identify the source of the leak, especially when third-party code is involved. However, he also discovered that this advanced detection mode significantly slowed down application performance, reducing throughput from 200 MB/sec to 25 MB/sec, and noted that the advanced mode could decelerate Netty by a factor of ten.