Fewer Logs, More Value

Honeycomb aimed to improve its internal telemetry by reducing noise in logs, focusing on deduplication and reduction strategies. They introduced the Log Deduplication processor in collaboration with ObservIQ, which aggregates duplicate logs and tracks how often they are deduplicated. This processor was integrated into the OpenTelemetry Collector's Contrib distribution, allowing for more efficient log management, particularly with Kubernetes logs. Initially, the deduplication did not meet expectations due to time variations in log records, but adjustments, such as parsing and ignoring time fields, improved results. Additionally, a custom Reduce processor was developed to merge similar logs with differing attributes, further lowering log volume. These efforts led to significant log reductions, up to 95% for some applications, underscoring the importance of proper log parsing and the utility of custom processors in managing telemetry data effectively.