What GitHub Webhook Latency Actually Looks Like

The analysis of GitHub webhook delivery latency reveals the complexities and challenges associated with measuring and managing webhook timing effectively. The team behind Mergify examined their webhook delivery process to identify latency issues, particularly during incidents when the latency increases significantly. They discovered that the lack of a "webhook delivered at" timestamp in GitHub's payloads necessitated the creation of their own metrics to accurately measure latency, leading to the development of a detailed dispatch table for mapping timestamps based on event types and actions. As they monitored latency data, they identified and corrected several timestamp mapping errors that skewed their measurements. The investigation highlighted that a significant portion of perceived latency was due to external factors, such as third-party CI systems like Jenkins, rather than GitHub itself. Despite achieving a more predictable latency measurement system, the team emphasizes the need for GitHub to provide better sender-side metrics and observability to fully understand and manage webhook delivery performance and meet their service level objectives.