Company
Date Published
Author
Alexey Milovidov
Word count
7160
Language
English
Hacker News points
None

Summary

This blog post discusses the optimization of LZ4 decompression speed in ClickHouse, a column-oriented database management system. The authors explore various techniques to accelerate decompression, including copying data in larger blocks (16 bytes instead of 8), using shuffle instructions for vectorized operations, and removing unnecessary checks. They also present a novel approach called "multi-armed bandits" that uses statistical sampling to select the best algorithm based on real-world data. The results show an average acceleration of 12-20% on modern x86 processors, with some datasets experiencing more than doubling in performance. However, the usefulness of these optimizations is limited in practice, and other factors such as dataset format and compression ratio play a significant role. The authors conclude that while LZ4 decompression can be accelerated, further research is needed to optimize algorithms for specific use cases.