How Honeycomb Uses Honeycomb, Part 6: Instrumenting a Production Service

In this continuation of the dogfooding series, the post explores how Honeycomb configures libhoney-go and adds instrumentation to its production services, using its API server for event receipt as an example. It outlines the process of augmenting each event with host and environment data, using dynamic fields to capture real-time metrics like goroutines and memory usage, and employing dynamic sampling to manage traffic effectively. The post emphasizes the use of a builder in the context of requests to facilitate the addition of fields to events and to support batch processing, highlighting the flexibility and reusability this method offers. Through practical examples, it demonstrates error handling, timing functions, and the benefits of wrapping third-party integrations, while inviting users to share their experiences and encouraging new users to try Honeycomb for enhanced systems observability.