GitHub Availability Report: April 2022

In April, GitHub faced three significant incidents affecting the availability of Codespaces and GitHub Packages, with each incident lasting several hours. The first incident on April 1 was caused by expired secrets that led to resource exhaustion in the US West region, spreading to other regions due to communication delays and limited access for secret rotation. To prevent future occurrences, GitHub has implemented resource monitoring and is transitioning away from dependency on secrets. On April 14, an investigation into contributing factors was initiated, with further updates promised in a future report. The third incident on April 25 involved increased CPU utilization in a GitHub Packages Registry database, caused by an unexpectedly high volume of "Create Manifest" commands. This led to internal server errors and user impact, which was mitigated by blocking the problematic activity and decoupling package listings from repository home pages. GitHub is addressing these issues through improved alerting, rate limiting, and performance fixes, while keeping users informed via their status page and engineering blog.