Company
Date Published
Author
Vadim Korolik
Word count
1628
Language
-
Hacker News points
None

Summary

OpenTelemetry (OTeL) is emerging as a standard for observability, offering a unified method to collect, process, and export telemetry data, including traces, logs, and metrics, with metrics providing key insights into system performance. Highlight.io, an open-source monitoring platform, has integrated support for OTeL metrics ingestion using ClickHouse, a high-performance columnar database, to enable scalable and cost-effective metric storage and querying. The implementation involves structuring a robust OpenTelemetry metrics pipeline incorporating ingestion, aggregation, querying, and visualization, with the use of Apache Kafka for buffering data and ClickHouse for storing it. The OpenTelemetry Collector acts as the data entry point, supporting multiple data formats and receivers, and enables efficient data processing before exporting to Highlight's API. The high-cardinality of metrics necessitates efficient aggregation to manage storage size, for which ClickHouse provides materialized views and TTL-based rollups. The system aggregates metrics into one-second intervals to balance granularity and storage efficiency, allowing for high-performance querying and visualization. This setup offers organizations deep insights into their applications with minimal storage and performance overhead, showcasing the potential of open-source observability solutions like Highlight.io.