July 2021 Summaries

The Red Bull Racing team faced a significant challenge during the Hungary 2020 Grand Prix when their car was damaged on the way to the grid. Instead of taking the car back to the garage and facing a penalty, they decided to fix it from the starting grid faster than ever before. This incident serves as an excellent example of effective incident response in high-pressure situations. The team quickly assembled, communicated effectively, adapted their procedures, and ultimately managed to repair the car within 18 minutes, allowing them to place second overall in the race.

In this interview with Colm Doyle, Incident Commander at Slack, he shares his journey into incident management and the role of an Incident Commander. He talks about how he got involved in incident response at Slack, his experience running incidents, dealing with stress during major outages, and the training process for new ICs. The interview also covers the structure of the Incident Commander rotation at Slack, the first 5 minutes after being paged, and challenges faced by Colm as an IC. Lastly, he shares his thoughts on diversifying the incident commander bench and a song he would choose to have in the background during an incident.

Incident.io experienced an issue with their incident identifiers appearing to jump unexpectedly due to Postgres sequences running ahead of the number of records in the database. The company initially suspected transaction rollbacks as a cause, but later discovered that follower nodes can see the state "that would prevail on the primary if it were to crash", which led to sequence values jumping up to 32 numbers forward during a database promotion process after an upgrade. To fix this issue, they implemented a trigger that runs before insert and looks for the maximum external ID value in the incidents table for a given organization, then increments it by one.

An incident refers to a problem that requires urgent action and may have negative impacts on the product, business or customers. It often necessitates coordination between multiple people or departments, communication with stakeholders, and post-incident review for learning purposes. Examples of incidents include insufficient staffing leading to increased delivery times, a major customer threatening to leave, security breaches, data leaks, recurring payment errors, etc. However, minor issues like CSS formatting problems, employee resignations, or isolated accidents are not considered incidents. Lowering the threshold for incidents can provide valuable learning opportunities and improve systemic responses to larger issues.

The text argues that having more incidents is not necessarily a bad thing, as it provides valuable information for learning and improvement. It suggests that organizations should aim for a healthy incident culture where incidents are seen as an acceptable cost of doing business. By embracing incidents and making them more visible, organizations can gain insights into their systems' real behavior and use this knowledge to make better decisions in the future.