March 2021 Summaries

Monitoring and observability in the current technological landscape present challenges due to fragmented tools and varied expectations, leading to complexity and steep learning curves for users. OpenTelemetry has emerged as a unifying force, allowing for a more cohesive ecosystem, yet users still face difficulties navigating multiple tools with disparate mechanics. Honeycomb offers a solution by integrating logging and tracing, where spans and events are synchronized to provide comprehensive insights. Span Events in Honeycomb, which evolved from the concept of Span Logs, allow detailed contextual information to be attached to traces, enhancing the ability to diagnose issues quickly. The platform's flexibility in handling diverse data, along with its powerful query capabilities, offers significant advantages over traditional logging tools, promoting efficiency and reducing tool fatigue. This innovative approach encourages the unification of traces and logs, making it easier for teams to access and analyze critical data seamlessly.

Honeycomb has introduced the INGEST_TIMESTAMP() derived column function to help users identify event latency issues and inaccurate timestamps, which can significantly impact event data analysis. A notable portion of events received by Honeycomb are delayed, either due to technical bottlenecks or incorrect system clocks, sometimes even arriving with future timestamps. By running repeatable COUNT queries in absolute time mode, users can observe discrepancies in event data availability due to events being delayed in transmission, buffered in pipelines, or impacted by proxies like Refinery. The article highlights an internal example where a tracing dataset revealed a maximum latency of 15 minutes, attributed to a specific span in their code running on AWS Lambda, which was delayed due to background thread issues. This latency issue was identified through the INGEST_TIMESTAMP() function, emphasizing its utility in uncovering both simple and complex latency issues in data pipelines. Honeycomb encourages users to investigate their event latency and offers support in resolving timestamp discrepancies.

Refinery, a platform-agnostic sampling solution from Honeycomb, is now generally available, allowing users to effectively sample telemetry data to manage costs without needing to sample data to use Honeycomb itself. By running Refinery locally with Docker, users can configure their applications to send data through Refinery before reaching Honeycomb, enabling the validation of sampling approaches with dry runs and the measurement of sampling impact with Honeycomb's Usage Center. Various sampling methods, such as deterministic, rules-based, and exponential moving average (EMA) sampling, are discussed, each offering unique benefits for managing different traffic patterns and ensuring efficient data handling. Users can experiment with Refinery's sampling strategies to maintain control over telemetry data while ensuring statistical accuracy in reconstructing traffic shapes. Honeycomb's Enterprise tier includes Refinery support, and a free 30-day trial is available for users interested in applying these techniques to their production workloads.

Megan Gleason has been promoted to Vice President of Product at Honeycomb, marking her first formal executive role after previously serving as Head of Product. Prior to Honeycomb, Megan held director-level positions at Auth0 and Chef Software. Her promotion reflects Honeycomb's commitment to internal growth and transparency in recognizing the contributions of its team members. Since joining Honeycomb in 2019, Megan has played a pivotal role in bridging gaps between engineering, sales, and marketing, enhancing product management, and developing a comprehensive product strategy that informed key business decisions. Her leadership has been instrumental in achieving significant milestones, such as advancing funding rounds and fostering a culture that values work-life balance and strategic collaboration. Megan's ability to listen, provide constructive feedback, and maintain a balance between innovation and stability has been crucial in driving the company's vision forward. Her promotion signifies a recognition of her contributions and the strategic importance of product leadership within the company.

Honeycomb has introduced Refinery, a new solution designed to help enterprises manage observability data at scale, offering a high-fidelity debugging experience while controlling costs. Refinery enables organizations to capture all telemetry data but selectively observe events that reflect significant system changes, reducing unnecessary data volume and improving system reliability. This tool is part of Honeycomb’s suite of enterprise offerings, which includes Service-Level Objectives (SLOs), Secure Tenancy, and extensive support and training services, aimed at enhancing production service management skills and fostering quicker troubleshooting of complex software applications. Refinery operates within customer infrastructure, granting full control over data sent to Honeycomb, and integrates seamlessly with Secure Tenancy and OpenTelemetry, addressing both data privacy and interoperability concerns. This approach democratizes observability tools traditionally limited to elite companies, enabling broader enterprise adoption by balancing data needs with cost considerations.

Refinery, now generally available, is a sampling solution by Honeycomb designed to optimize resource costs and maintain data fidelity at scale, particularly for enterprises handling tens of billions of events monthly. By enabling users to focus on critical debugging data and discard unnecessary information, Refinery helps manage the immense data volumes that can otherwise overwhelm systems and budgets. It supports dynamic and tail sampling methods, allowing for flexible data management and cost control, with features such as dry-run mode and enterprise support included in Honeycomb's plans. Refinery, operating on user infrastructure, reduces network egress traffic and enhances observability without compromising data quality, as evidenced by user testimonials from companies like GOAT Group and Geckoboard. Honeycomb's broader mission aims to democratize production excellence, offering features like Debuggable Service-Level Objectives and secure data handling to all organizations, not just elite ones, and includes options for exclusive training and support to facilitate widespread adoption of observability practices.