Spooky Tales of Testing In Production: A Recap and Lessons Learned

In a review of a SPOOKY HALLOWEEN meetup cohosted by LaunchDarkly, several talks on incident response strategies highlighted the importance of effective communication and prioritization during crises. Paul Biggar's discussion emphasized the necessity of assigning a leader to manage incidents and stressed transparency with customers as a key factor in retaining them post-crisis. Marc Deven illustrated the value of transparency and business insurance during a glitch at a company that mistakenly priced products at a penny, emphasizing how effective incident management can become a point of pride and influence company culture. Eric Pollmann from Google shared insights into handling a data push disaster, underscoring the need for robust revert procedures and the realization that even the best systems require hands-on problem-solving. The meetup also underscored the benefits of using gating software, like LaunchDarkly, to manage risks associated with software releases, and the collective learning from these incidents was seen as essential for improving incident response protocols at Honeycomb.