GitHub Availability Report: August 2023
Blog post from GitHub
In August, GitHub experienced two significant incidents that impacted service performance, starting with a delay in webhook processing on August 15 due to a spike in deliveries, which was mitigated by blocking event sources to clear the backlog. As a result, improvements were implemented to handle higher traffic and manage load sources more effectively. On August 29, a separate incident caused delays in background job processing due to issues with a Kafka consumer group, which was resolved by shifting the load to a standby service and redeploying the primary service. Despite the delays, no data was lost, and GitHub has extended monitoring to prevent future occurrences while encouraging users to check their status page and engineering blog for updates.