Company
Date Published
Author
Vadim Korolik
Word count
1743
Language
-
Hacker News points
None

Summary

Highlight.io, an open-source monitoring distribution, faced scaling issues with its ClickHouse setup after onboarding larger customers, leading to the need for optimization strategies to handle the massive data ingestion efficiently. The company dealt with challenges like high CPU loads and inefficient query performance due to small data batch inserts and numerous data "parts" created during storage. To address these issues, they implemented several strategies, including using larger batch inserts, optimizing data storage formats to avoid costly conversions, and adjusting the order of data to reduce unnecessary merges. They also explored the use of materialized views over projections for more reliable data querying and set up TTLs to manage active data parts. These efforts resulted in a more efficient, cost-effective, and real-time data processing experience for their customers.