February 2024 Summaries

In this text, it is discussed that noting follow-up actions after an incident response process is crucial but often overlooked or forgotten. The introduction of a new feature called "Suggested Follow-ups" aims to address this issue. Rob, the project lead for this feature, shares insights into its development, including timelines and challenges faced by the team. An AI announcement episode featuring Rob's discussion can be accessed through the provided link.

Effectively managing an on-call rota is crucial for ensuring the round-the-clock availability of services and protecting the employees who make it possible. Some best practices include using software to automate scheduling, setting up teams with clearly defined responsibilities, establishing escalation policies, defining time limits for issue resolution, enabling easy overrides for cover, maintaining high levels of transparency and communication among team members, and providing training through shadow shifts. By implementing these tactics, organizations can improve team accountability, service reliability, and customer satisfaction.

The author joined incident.io as VP of Engineering due to their interest in solving unique challenges that arise during emergencies. They were impressed by the company's team, problem space, product, and pace. The team was highly collaborative, intelligent, and customer-focused. The problem space involves dealing with incidents and learning from them. The author has been a fan of the product since its early days and appreciates its rapid evolution and high quality features. Lastly, they are excited about the company's commitment to excellence and innovation in the incident management space.

Incident management can be challenging for many teams due to its stressful and unoptimized nature, often resulting in longer response times and lack of preventive actions. This not only affects the responders but also negatively impacts customers. However, improving incident response is possible by implementing a good process. In this episode of The Debrief, experts Kerim Satirli and Lawrence Jones discuss their experiences with incidents and provide insights on how to make incident response less painful for teams.

In a conversation with Kerim Satirl, Senior Developer Advocate at HashiCorp, the importance of having a good incident management process was highlighted. According to Satirl, incidents should be managed once every two weeks on average, covering security issues and unfinished commits. A past incident involving a paused database migration emphasized the need for proper tools and processes in incident response. The lack thereof can lead to rushed decisions that exacerbate problems. Satirl also discussed why startups and startup-like organizations often neglect prioritizing incident response, attributing it to cost-saving measures like minimal documentation and testing. However, this approach can backfire when incidents occur, leading to an all-hands-on deck situation where the root cause is difficult to identify due to insufficient documentation. The value of being involved in incident response lies in the ability to fix problems one has built and ensuring that proper tooling is available for efficient communication and understanding of the issue's severity. This allows engineers to focus on solving the problem while others can gather information independently, ultimately leading to a more effective resolution process.

The text discusses a survey conducted among world-class organizations such as Etsy, SumUp, and Ramp to understand how they measure their incident management efficiency. MTTx metrics, particularly DORA's MTTR, are the most common metrics used by these organizations. However, there is a disconnect between upper management and teams regarding which metrics to track. While many teams take action on insights from their metrics, some do not due to time constraints or lack of usefulness. Additionally, several responders expressed interest in tracking cost or financial impact as it relates to incidents.

In a recent development, an AI assistant has been created that can automatically provide various useful incident response data points upon request. This feature is called Assistant and is part of a new product powered by AI technology. The project lead for Assistant, Charlie Revett, shares insights into the process of building this innovative tool, including timelines, challenges faced, and learning prompt engineering while developing it. For more information on Assistant, listen to the related podcast episode or read the blog post about it.