The Gorilla in the Room: Exploring RedisTimeSeries Performance Optimizations

RedisTimeSeries is a time-series database module that stores large amounts of streaming data in-memory, allowing for fast processing thanks to the Gorilla compression/decompression algorithm. The algorithm achieves a 10X compression ratio and was developed by Meta for its large-scale and distributed Time-Series Database (TSDB). Intel and Redis are collaborating to investigate potential performance optimizations for the RedisTimeSeries database. They have explored vectorizing the compression/decompression computation, but found that it would not pay off in terms of performance and code complexity. Instead, they focused on optimizing the decompression routine for floating-point values and discovered that using a dictionary and/or run-length encoding could lead to better performance and/or compression ratio. The team also considered alternative approaches, such as storing data as integers instead of floating points, but concluded that Gorilla remains the best compression ratio option.