Tail sampling with OpenTelemetry and New Relic

Tail sampling is a method used in distributed systems to manage observability costs and focus on important data by sampling traces after all spans in a request are completed, contrasting with head-based sampling, which occurs at the beginning of a request. This approach allows for filtering based on specific criteria, such as error traces or latency, reducing the amount of data stored and processed. Implementing tail sampling in OpenTelemetry involves using the tail sampling processor, which relies on policies to determine which traces to sample, though it presents challenges such as unpredictable costs and the need for a suitable collector setup. New Relic's Infinite Tracing offers an alternative solution by simplifying the process through a trace observer that supports OpenTelemetry and New Relic's APM agents, enabling users to configure sampling rules without altering their existing infrastructure. Despite its advantages, tail sampling requires careful consideration of its limitations and potential issues, including resource allocation and the possibility of fragmented traces due to incomplete data collection.