Company
Date Published
Author
Jamie Tanna
Word count
2508
Language
English
Hacker News points
None

Summary

In October 2024, Elastic inadvertently made their most critical public GitHub repositories private, causing a significant service disruption for their customers and open-source users. The incident occurred during an effort to enhance internal source code security by migrating repositories from internal to private visibility. However, a lack of validation in their automation scripts led to 63 public repositories, including Elasticsearch and Kibana, being incorrectly marked as private. The incident highlighted the importance of verifying real-world states before executing automated changes, breaking down large-scale changes into smaller iterations, and decentralizing authority for sensitive actions. Elastic's well-practiced incident management process, collaboration with GitHub, and a blameless culture played crucial roles in resolving the issue within seven hours and restoring the repositories' visibility. The incident prompted Elastic to implement stricter access controls and inventory management to prevent similar occurrences in the future.