OpenTelemetry Gotchas: Phantom Spans

Ian Duncan, a Staff Engineer at Mercury, shares insights into addressing a tracing issue with OpenTelemetry in Haskell, where unrelated web requests were mistakenly linked in the same trace. OpenTelemetry, a framework for logging, metrics, and distributed tracing, allows developers to instrument code using a single API, sending data to various backends like Honeycomb, which Mercury primarily uses. The core of the issue was traced back to external requests misusing W3C Trace Context headers, causing them to be incorrectly identified as part of the same trace. By removing these headers at the Cloudflare level, Mercury resolved the problem and prevented potential denial-of-service attacks on their tracing infrastructure. This experience highlights the challenges and solutions in managing distributed tracing in a complex system, emphasizing the importance of understanding and correctly implementing tracing propagators.