January 2025 Summaries

Highlight.io, an open-source monitoring platform, has revamped its onboarding process to enhance user experience by shifting from product-specific tutorials to a language-specific approach. Previously, users had to integrate individual products like Session Replay, Logs, and Traces sequentially, which often led to some features being overlooked. This new onboarding flow allows developers to integrate their preferred language, such as JavaScript, Python, or Go, and immediately receive all relevant observability data without additional setup steps. By navigating to app.highlight.io/connect, users can select a language or framework, and the platform automatically fetches and displays relevant setup guides and code blocks, streamlining the integration and enabling immediate access to the full suite of Highlight's tools. The company is committed to continuously improving this process and welcomes user feedback through its communication channels.

Integrating Amazon Kinesis Data Firehose with the OpenTelemetry Collector enhances the observability of real-time streaming data by allowing logs and metrics to be exported to various destinations like AWS services or other platforms. Highlight.io, an open-source monitoring platform, supports the Firehose format through its cloud-hosted OpenTelemetry collector, enabling users to manage data export without additional infrastructure. The OpenTelemetry Collector, a vendor-neutral proxy, is equipped to process and export telemetry data using standardized receivers, reducing engineering overhead by eliminating the need for custom parsing and transformation. Receivers for AWS Firehose, such as those in the opentelemetry-collector-contrib repository, can handle specific data formats, ensuring compatibility with OpenTelemetry's processing pipeline. The configuration involves setting up Firehose receivers to listen on different ports for logs and metrics, demonstrating how to leverage existing OpenTelemetry receivers for efficient data collection and integration with observability tools. Highlight.io offers users a convenient solution to manage these integrations by providing pre-configured setups that reduce complexity and enhance data visibility.

The blog post explores the trade-offs between physical and logical data isolation methods in multitenant applications, particularly focusing on how Highlight.io, an open-source monitoring platform, addresses these challenges using a multitenant ClickHouse cluster. Physical isolation involves maintaining separate infrastructure for each tenant, which can be resource-intensive, while logical isolation uses shared infrastructure with application-level access restrictions, which can be prone to errors and vulnerabilities. Highlight.io employs ClickHouse row policies to enforce logical data isolation, creating roles and row policies for tenants to ensure secure access to data without the complexity of managing numerous roles and policies. They implemented a hybrid approach that leverages ClickHouse's row policies and custom settings, using a context object in the clickhouse-go Golang driver to enhance security against SQL injection and ensure tenant-level isolation. This method reduces error risk, avoids the overhead of creating extensive access control objects, and provides a more secure solution for managing data in a multitenant environment.

OpenTelemetry (OTeL) is emerging as a standard for observability, offering a unified method to collect, process, and export telemetry data, including traces, logs, and metrics, with metrics providing key insights into system performance. Highlight.io, an open-source monitoring platform, has integrated support for OTeL metrics ingestion using ClickHouse, a high-performance columnar database, to enable scalable and cost-effective metric storage and querying. The implementation involves structuring a robust OpenTelemetry metrics pipeline incorporating ingestion, aggregation, querying, and visualization, with the use of Apache Kafka for buffering data and ClickHouse for storing it. The OpenTelemetry Collector acts as the data entry point, supporting multiple data formats and receivers, and enables efficient data processing before exporting to Highlight's API. The high-cardinality of metrics necessitates efficient aggregation to manage storage size, for which ClickHouse provides materialized views and TTL-based rollups. The system aggregates metrics into one-second intervals to balance granularity and storage efficiency, allowing for high-performance querying and visualization. This setup offers organizations deep insights into their applications with minimal storage and performance overhead, showcasing the potential of open-source observability solutions like Highlight.io.

OpenTelemetry is an open-source observability framework designed to help developers monitor and troubleshoot distributed systems by providing standardized data formats and integration points for collecting, processing, and exporting telemetry data such as traces, metrics, and logs. The guide explores the integration of OpenTelemetry with React Native applications, detailing how to instrument apps to send telemetry data to a backend of choice, specifically using Highlight.io. Key components include setting up a provider for configuration, creating a custom exporter to handle data transmission, and utilizing a processor for pre-processing tasks like batching. The guide further demonstrates how to implement tracing, logging, and error handling by creating custom functions and monkeypatching JavaScript's console methods to ensure seamless data flow to Highlight. Additionally, the guide highlights the flexibility of OpenTelemetry in adapting to different application requirements, encouraging developers to customize it as needed.

OpenTelemetry is a versatile, vendor-agnostic specification for sending telemetry data to observability platforms like Highlight.io and Grafana, offering comprehensive support for traces, metrics, and logs. This guide provides an in-depth look at implementing OpenTelemetry in Python, detailing how to instrument applications to collect and send telemetry data to a chosen backend. Key components include providers, processors, and exporters, which facilitate the configuration and transmission of telemetry data. The guide emphasizes both manual and auto-instrumentation techniques, such as middleware integration in applications like FastAPI, to streamline monitoring and debugging processes. It also offers practical examples, including setting up OpenTelemetry resources in a Flask application, to demonstrate the configuration and use of these components.