October 2023 Summaries

Highlight transitioned from using OpenSearch to Clickhouse for its search backend to improve performance and reduce costs. Originally, Highlight used OpenSearch to handle metadata searches across sessions and errors, but as customer data grew, performance declined due to increased search times and memory dependency. Clickhouse, being a column-oriented database, allows more efficient querying by loading only necessary data columns, which is advantageous for complex searches that can't benefit from indexing. The shift to Clickhouse also enabled the use of materialized views to enhance field autocomplete suggestions by tracking field usage frequency. To maintain data consistency between Postgres, the primary data store, and Clickhouse, Highlight implemented a system using Kafka, ensuring eventual consistency despite potential update delays. This migration resulted in significantly faster search query times, reduced operational costs, and simplified architecture by phasing out the expensive OpenSearch cluster.

Effectively managing data ingestion in today's data-intensive environments is crucial for maintaining system efficiency and cost-effectiveness. Advanced ingestion filtering techniques, such as sampling, rate limiting, and exclusion queries, offer powerful tools for controlling the volume and type of data recorded in monitoring and analytics platforms. Sampling involves storing only a subset of incoming data points using probabilistic methods, which helps reduce storage costs and focus on overarching patterns rather than individual data points. Rate limiting prevents system overloads by discarding excess data when a set threshold is reached within a specific time frame, which is particularly useful during unexpected spikes in usage. Exclusion queries provide granular control by allowing specific conditions under which data should not be ingested, such as filtering out data from certain environments or geographies. These techniques directly impact billing, as only the data that is retained is charged, leading to potential cost savings. Balancing cost savings with the retention of meaningful data is essential, and regular reviews and adjustments to these filters are recommended to align with evolving requirements and ensure effective data analysis.

Highlight has introduced a new Tracing product, expanding its use of telemetry data beyond logging to include detailed code execution insights, integrated with session replay, errors, and logs. This enhancement, built on OpenTelemetry and Clickhouse, provides developers with greater visibility into application performance by adding instrumentation to key code paths, facilitating the identification of codebase hotspots, errors, and performance bottlenecks. The tracing feature allows users to analyze API handlers and other components to understand execution times and errors, enabling data-driven decisions to optimize performance and user experience. Highlight plans to further develop telemetry capabilities, with future updates expected to include metrics support, while currently encouraging users to explore and provide feedback on the new Tracing functionality.

Highlight has introduced a "default" privacy mode in its SDK versions 8.0.0 and later, aiming to enhance data protection by obfuscating personal identifiable information (PII) using regex expressions. This mode intelligently hides data like phone numbers, social security numbers, and email addresses, distinguishing it from the previous "strict" privacy mode which indiscriminately obfuscated all text. While the default mode effectively manages static text, it faces challenges with dynamic inputs, sometimes delaying recognition until enough data is entered. The mode may also over-obfuscate data or miss elements broken by HTML tags due to context limitations, yet it represents a significant advancement in safeguarding customer data for companies wary of data exposure. Highlight plans to continue refining this privacy feature and welcomes feedback through their Discord channel.

An open-source session replay tool developed by highlight.io utilizes rrweb to analyze user interactions and performance regressions within web applications while maintaining minimal impact on user experience. Performance testing revealed that enabling session replay resulted in slight increases in browser resource consumption and interaction latency, yet these changes were negligible enough not to affect user perception adversely. The study, conducted using a React application and Node.js automation, found that while session replay increases memory usage and CPU activity, it does not significantly impact responsiveness, even in scenarios with intensive DOM interactions. The optimal implementation in modern browsers aids in efficiently managing DOM changes, with future goals including profiling the Mutation Observer API to better understand the sources of overhead. The experiment encourages further community engagement and benchmarking across different hardware configurations.

Code generators significantly enhance development workflows by saving time, ensuring standardization, and promoting consistency across the codebase. They facilitate efficient onboarding, maintain a historical record of coding decisions, and can be used to generate various elements such as backend boilerplate, data-fetching hooks, React components, and support files. By automating the creation of these elements, code generation streamlines the development process and improves code quality. Although AI tools like GitHub Copilot offer additional efficiency improvements, they lack the speed and consistency provided by dedicated code generation tools. The Highlight team encourages others to share their experiences with code generators and offers their open-source codebase as an example of effective use.

Highlight, a platform providing full-stack visibility into application errors, has implemented a new feature using a language learning model to effectively distinguish between significant and inconsequential errors. The system employs a 1024-dimension open-source model to tag errors, such as "authentication error" or "database error," and group similar errors, even if they have different stack traces, by using Euclidean distance between their embeddings. This approach, which leverages the OpenAI embeddings API and is hosted on Hugging Face, allows developers to quickly identify and address issues by reducing noise and focusing on actionable errors. The first version of this error grouping logic has been integrated into their cloud product, and the Highlight team is open to further suggestions from the community for expanding the use of LLM tooling in observability.

GitHub enhanced stacktraces now offer developers the ability to see detailed context for backend errors, similar to what sourcemaps provide for frontend errors, by utilizing GitHub repositories to retrieve and map the corresponding code files. This feature is facilitated by Highlight, which requires some initial configuration to map file paths correctly during the build process, as backend languages often involve compiled code that obscures direct mappings. The system minimizes GitHub API rate limitations by caching fetched files in S3 using a Git Sha commit hash, ensuring that only the latest code changes are referenced. Although providing the serviceVersion field to the Highlight SDK can enhance accuracy, the feature will still function by fetching the latest commit hash from the default branch if this field is absent. Future plans aim to incorporate more detailed features like file change history and linking errors to specific commits, enhancing the debugging process and providing more comprehensive error context.

Vercel Edge Runtime, built on Cloudflare Workers, offers a compelling solution for serverless computing by eliminating cold starts and reducing network latency, thanks to its global data center deployment and cost-effective infrastructure. However, its lack of Node.js compatibility can pose challenges, as developers may encounter unsupported API errors during deployment. Highlight supports the Edge Runtime by utilizing a customized version of the opentelemetry-sdk-workers package, addressing compatibility issues with Edge-incompatible API calls through strategic patching in the CI/CD pipeline. The Highlight team is exploring ways to simplify API endpoint wrapping in @highlight-run/next, considering both automated approaches using Rollup and static analysis, and a codemod for direct codebase edits, though these solutions require careful validation to ensure maturity and stability.