December 2022 Summaries

As 2022 concludes, Honeycomb is engaging its community in a festive initiative called #o11ydaytreats, offering participants a chance to earn swag and enter a draw for a signed copy of "Observability Engineering" or a strategy session with Charity Majors or Liz Fong-Jones. Participants can complete various tasks related to Honeycomb's observability tools, such as using the Happy o11ydays app for chart art, exploring the Sandbox tour for debugging workflows, or integrating services like Datadog and AWS with Honeycomb. By sharing their experiences with these tasks on social media using the hashtag #o11ydaytreats, users increase their chances of winning. The initiative aims to encourage users to expand their observability setup, engage with the community, and prepare for the new year with enhanced insights and strategies.

Honeycomb introduces a holiday-themed release featuring OpenTelemetry enhancements, including the availability of Honeycomb Metrics for Free users, leveraging the stabilization of Metrics in various OpenTelemetry SDKs. This advancement allows users to gather and analyze application metrics using instruments like Counters and Histograms, enhancing the contextual understanding of applications when combined with OpenTelemetry Tracing. Honeycomb has also launched an SDK distribution for Node.js, simplifying data transmission to Honeycomb with minimal configuration, offering features like multi-span attributes and local trace visualization. To handle high volumes of event data, Honeycomb has enhanced the OpenTelemetry Collector filterprocessor to filter unnecessary span events, ensuring efficient data management. The company encourages users to explore these updates and relax during the holidays, promising more innovations in the coming year.

A longstanding bug in Honeycomb's columnar datastore, present for over two years, unexpectedly caused data losses and query crashes, leading to an investigation led by Paul Osman. The issue emerged from a missing trailing slash in the segment lifecycle management process, which resulted in accidental deletions of data segments due to hash collisions in S3 object naming. This bug was particularly elusive because it was masked by Honeycomb's prefixing scheme, which was intended to optimize performance by avoiding hotspots. The discovery process involved analyzing S3 logs and custom instrumentation, revealing the necessity of having detailed observability tools to diagnose such complex issues. The fix was a simple code change, but it highlighted the importance of teamwork and the value of in-depth system instrumentation to uncover and resolve intricate bugs in large-scale data systems.