Company
Date Published
Author
Michael Malis
Word count
1605
Language
English
Hacker News points
None

Summary

Heap, a customer analytics tool, significantly improved its Postgres cluster's CPU efficiency by applying basic performance analysis techniques, leading to a tenfold enhancement and substantial cost savings. Initially, Heap's unique indexing strategy, which utilized Postgres' partial indexing for faster event retrieval, resulted in unexpectedly high CPU usage. This was attributed to the high number of regular expression evaluations required for each event insertion. However, further investigation using flame graphs revealed that the CPU load was due to frequent fetching and parsing of index metadata rather than predicate evaluations. By implementing batch inserts instead of individual ones, Heap reduced the need for repeated metadata processing, achieving a 10x reduction in CPU usage. This simple change not only improved ingestion throughput by approximately tenfold but also eliminated the need for additional CPU resources in the foreseeable future.