GitHub Availability Report: October 2022

In October, GitHub experienced four significant incidents impacting the availability of its services, with further insight provided into a September incident affecting Codespaces. On October 26, a major issue affected most Codespaces customers for nearly four hours, with ongoing investigations to determine the root cause. On October 13, a recent deployment led to increased error rates in the Projects API due to a database validation issue, resolved within 48 minutes through a rollback and subsequent testing improvements. The October 12 incident involved a global configuration change for Codespaces that caused a schema conflict, resulting in service degradation for over three hours before a complex rollback restored functionality. On October 5, a rapid influx of webhook events due to automated user activity led to a backlog and service delays, resolved by disabling the automated accounts and updating webhook delivery processes. An earlier incident on September 28 involved missed steps in a secret rotation checklist, causing traffic disruptions in Codespaces, which were mitigated by rerunning the missed steps and enhancing monitoring and processes for future secret rotations.