How We Made Payload Search 60x Faster in ClickHouse

At Hookdeck, a platform that processes hundreds of millions of webhooks daily, a significant enhancement was achieved in payload search speed using ClickHouse, resulting in a 60-fold increase in performance, reducing average search latency to approximately 400 milliseconds. Initially, as data volumes grew, search queries began timing out, hampering a critical debugging tool for users. The improvement was realized through two key techniques: hash-bucketed payload storage and variable-window iterative scanning. The first method involved distributing data across multiple bucketed columns based on hashed values, reducing the amount of data scanned per query. The second technique involved issuing queries with expanding time windows until enough results were gathered, significantly reducing the number of rows scanned. This approach was complemented by adaptive granularity adjustments and probe queries to optimize scanning efficiency. These strategies collectively improved reliability, eliminated timeout-related failures, and transformed user experience by delivering faster, incremental search results.