Company
Date Published
Author
Jakub Oleksy
Word count
332
Language
English
Hacker News points
None

Summary

In June, GitHub experienced two incidents that affected its services' performance. On June 5, a service misconfiguration due to a secret rotation initiative led to a degradation in the GitHub Issues service, preventing project-related events from displaying on issue timelines for 142 minutes. The problem was traced back to old expired secrets being used, and it was resolved by correcting the service configuration, with future incidents expected to be avoided through this streamlined setup. On June 27, an invalid infrastructure credential caused all in-progress migrations to fail for 58 minutes, prompting a pause in new migrations to prevent further issues. The incident was resolved through manual intervention by first responders, and queued migrations resumed successfully. GitHub plans to address gaps in monitoring and alerting for infrastructure credentials to prevent similar occurrences, and users are encouraged to follow their status page and engineering blog for updates.