June 2018 Summaries

Chaos Engineering is a method used to test and improve the resiliency of systems by intentionally introducing failures to identify weaknesses and enhance recovery mechanisms. A recent incident at Gremlin highlighted the importance of this approach when an unattended chaos experiment led to a network misconfiguration, initially mistaken for a DNS outage. By iterating on chaos experiments, Gremlin identified the need to upgrade their API health checks to better detect database connectivity issues without overwhelming the system. This process involved implementing a "Dead Man's switch" to monitor failed connections and upgrading health checks to ensure they respond appropriately when database or cache issues occur. The incident emphasized the necessity of constantly evolving resiliency mechanisms and chaos experiments to address unforeseen failures, ultimately enhancing system stability and reliability.