Bounding ksqlDB Memory Usage

Managing memory consumption is critical for maintaining the stability and performance of services like ksqlDB, especially to prevent crashes and meet Service Level Agreements (SLAs). This blog post explores how ksqlDB users have faced challenges with memory limits, particularly in the Confluent Cloud environment, and describes the steps taken to address these issues. It discusses the configuration of memory limits, the role of JVM heap and off-heap memory in memory consumption, and the specific challenges encountered with RocksDB, a key component used by ksqlDB for storing state. The investigation revealed that memory consumption could exceed configured limits due to the way different threads utilize memory arenas, leading to memory not being released as expected. Profiling and testing strategies were employed to understand and reproduce the memory growth issue, leading to the discovery that swapping the memory allocator from glibc to jemalloc resolved the problem by preventing excess memory growth. The article concludes with recommendations for configuring ksqlDB to adhere to desired memory limits effectively, emphasizing the importance of using KsqlBoundedMemoryRocksDBConfigSetter and considering alternative allocators like jemalloc to manage memory usage better.