Alert Evaluations: Incremental Merges in ClickHouse

Highlight.io, an open-source app monitoring platform, tackled performance issues in their alert system by utilizing specific features of ClickHouse, a columnar database optimized for large datasets and real-time analytics. They faced challenges in efficiently processing large volumes of data for real-time alert evaluations, particularly with complex aggregate functions like calculating the p99 duration of network requests. By employing ClickHouse’s memory-efficient approximate algorithms, particularly the -State and -Merge functions, Highlight was able to compute and store intermediate results for incremental calculations, significantly improving processing speed and reducing memory usage. This approach allowed them to achieve a 10x speedup in alert evaluation time and drastically decrease memory consumption, demonstrating the effectiveness of ClickHouse's state and merge combinators in optimizing data processing tasks.