When GitHub Webhooks Lie: How an Empty Array Broke Our Merge Queue

GitHub webhooks can sometimes deliver payloads with stale data, leading to issues such as the unexpected ejection of pull requests from merge queues, as experienced by a team that traced the problem to out-of-order webhook delivery. The incident involved a structurally valid webhook payload with an empty labels array, despite the pull request having all necessary labels, illustrating how out-of-order delivery and incomplete payloads can corrupt database states when webhooks are treated as the authoritative source. The team identified that the webhook's action field should be used to determine which fields to trust, rather than assuming every field in every event is both authoritative and current. To address this, they implemented an action-aware update mechanism, ensuring that labels are only overwritten when the webhook action pertains to label changes, thus preserving existing labels in the database. This targeted fix, which was integrated into the model layer for broader applicability, underscores the importance of recognizing patterns of incomplete payloads and designing resilience into webhook processing systems.