Event Latency: What It Is and Why You Should Care

Honeycomb has introduced the INGEST_TIMESTAMP() derived column function to help users identify event latency issues and inaccurate timestamps, which can significantly impact event data analysis. A notable portion of events received by Honeycomb are delayed, either due to technical bottlenecks or incorrect system clocks, sometimes even arriving with future timestamps. By running repeatable COUNT queries in absolute time mode, users can observe discrepancies in event data availability due to events being delayed in transmission, buffered in pipelines, or impacted by proxies like Refinery. The article highlights an internal example where a tracing dataset revealed a maximum latency of 15 minutes, attributed to a specific span in their code running on AWS Lambda, which was delayed due to background thread issues. This latency issue was identified through the INGEST_TIMESTAMP() function, emphasizing its utility in uncovering both simple and complex latency issues in data pipelines. Honeycomb encourages users to investigate their event latency and offers support in resolving timestamp discrepancies.