Interview with Honeycomb Engineer Chris Toshok: Dogfooding OpenTelemetry

Honeycomb leverages its own platform to test and refine UX changes by using OpenTelemetry (OTel) alongside its existing Beeline SDK in its Poodle frontend service, which is responsible for rendering visualizations. Engineer Chris Toshok undertook the integration of OTel with Poodle to examine schema compatibility and performance differences, opting to double-send data to separate datasets to maintain existing functionalities and observe discrepancies. Toshok noted that Beeline's API is more closely aligned with Honeycomb's features, while OTel operates at a more general layer, lacking some Honeycomb-specific capabilities like trace-level attributes. Despite minor API differences and the need for type manipulation, the overall structure of instrumentation remained consistent, and performance remained unaffected after implementing OTel, highlighting the potential for seamless integration and evaluation of observability tools.