January 2024 Summaries

During a recent product update, new visualization tools such as Flame Graphs and Integrated Session Views were introduced to enhance trace analysis in application monitoring. These tools aim to address the challenge of extracting actionable insights from traces by allowing users to efficiently analyze performance bottlenecks and view detailed trace data within network request sessions. Flame Graphs provide a visual representation of where time is spent in a trace, enabling quick identification of performance issues. The integration with OpenTelemetry allows for seamless navigation from user session replays to specific database calls, such as identifying a missing SQL index without leaving the session. Additionally, advanced filtering options and high-level metric visualization offer users enhanced trace accessibility and overview capabilities. For more technical insights, an in-depth post on Materialized ClickHouse Views is available, detailing how these features support Highlight Tracing.

Highlight has significantly enhanced its Query Builder by introducing a proprietary query language to better manage the large amounts of observability data it handles daily. This new language, which can be executed on both client and server sides, facilitates complex query structures with advanced numeric filtering and logical operations such as NOT and OR, thereby improving performance and setting the stage for future developments. The update also includes client-side query validation, which provides instant feedback to users for more accurate query construction, and an autocomplete feature that simplifies the use of operators like =, !=, >, <, <=, and >=. Syntax highlighting has been improved to make even intricate queries easier to read and understand, enhancing user experience by making query construction more intuitive and troubleshooting quicker.

Combining OpenTelemetry with Highlight offers a flexible and vendor-neutral solution for performance tracing, logging, metrics, and error monitoring without the risk of vendor lock-in. OpenTelemetry supports a wide range of libraries and languages, providing compatibility and versatility, while Highlight minimizes the engineering effort involved in storing and visualizing data. Highlight efficiently ingests data using Kafka and ClickHouse, providing a robust user interface for tracing and logging, and allowing the correlation of backend data with frontend user actions. This integration simplifies data management, enabling developers to focus on optimizing application performance.

Highlight.io, an open-source observability solution, adopted ClickHouse to manage and query large volumes of observability data, particularly for complex analytical tasks. ClickHouse, an OLAP database, is more suited for aggregate data analysis compared to PostgreSQL, which excels in OLTP tasks and single-record lookups. Materialized Views (MVs) in both databases help optimize query performance by storing transformed data. Highlight.io uses MVs in ClickHouse to efficiently handle various query patterns, such as aggregating traces and optimizing row lookups by different attributes, which is crucial for their tracing product's functionality. An example involves creating MVs to differentiate numeric and non-numeric trace attributes for quick searches, as well as optimizing trace searches by TraceId to support their UI needs, despite the additional resource consumption involved in maintaining these views.