October 2017 Summaries

In this continuation of the dogfooding series, the focus is on enhancing the efficiency of Honeycomb's query retrieval process by optimizing the polling intervals for fetching query results. Originally, the system relied on a 1-second polling interval, which often led to unnecessary delays despite the fast query processing capability of Honeycomb's columnar store. By analyzing query completion times, the team identified that many queries were completed and persisted in under 412 milliseconds, prompting a reduction in the initial polling interval to 250 milliseconds. This change significantly improved the speed at which users received query results, with 91% of queries being serviced in just one poll iteration and 94% in two, effectively reducing the end-to-end completion time by half a second for the majority of sampled queries. While this adjustment has yielded positive results, further optimizations in the frontend and transport systems are necessary to eliminate remaining inefficiencies and enhance overall user experience, reinforcing Honeycomb's commitment to future advancements in systems observability.

In this continuation of the dogfooding series, the post explores how Honeycomb configures libhoney-go and adds instrumentation to its production services, using its API server for event receipt as an example. It outlines the process of augmenting each event with host and environment data, using dynamic fields to capture real-time metrics like goroutines and memory usage, and employing dynamic sampling to manage traffic effectively. The post emphasizes the use of a builder in the context of requests to facilitate the addition of fields to events and to support batch processing, highlighting the flexibility and reusability this method offers. Through practical examples, it demonstrates error handling, timing functions, and the benefits of wrapping third-party integrations, while inviting users to share their experiences and encouraging new users to try Honeycomb for enhanced systems observability.