September 2021 Summaries

Incident.io has raised $4.7 million from Index Ventures, Point Nine, and a group of angel investors to accelerate the growth and development of its incident management platform. The company aims to provide world-class incident response for organizations by building a powerful command center within Slack. With the support of experienced engineers and customer feedback, Incident.io is refining its product to enable organization-wide adoption and improve overall incident response processes.

During an incident, it is crucial to build an understanding of what happened and why in order to fix the issue. However, one must be cautious not to rely on incorrect or unverified assumptions that can lead to misdiagnosis and ineffective solutions. In this particular case, a team initially assumed that Google Cloud Platform (GCP) was responsible for downcasing HTTP headers, but later discovered that an upgrade to HAProxy caused the issue. To avoid such mistakes, it is recommended to show your working by logging all actions taken during incident response, cross-check assumptions periodically, and be aware of personal biases that may influence decision-making.

Incident response can still be challenging despite experience, tooling, and preparation. Some common difficulties include determining the most highly leveraged role to play, getting up to speed without disrupting the flow, making quick decisions as an individual versus seeking consensus, monitoring parallel actions and investigations, striking a balance between gut feelings and evidence gathering, recovering from bad assumptions, and reassessing initial findings when incidents escalate. These challenges provide opportunities for improvement in incident response processes and tools.

Lawrence, a former Principal Engineer at GoCardless, has recently joined the company. He was responsible for building self-service infrastructure tooling and leading work on Open Banking. With experience in handling incidents as an SRE at a fintech firm, Lawrence is eager to contribute his expertise to help incident.io develop into a valuable product. In his leisure time, he enjoys cycling, crossfit, and discovering new food spots in the city. To learn more about him, visit his blog or follow him on Twitter at @lawrjones.

Uma Chingunde is the Vice President of Engineering at Render, a company that provides fast app and website hosting services. She has worked in infrastructure for many years, starting her career at VMware before moving on to Stripe where she was responsible for managing core infrastructure components. Incident management became an important part of her role during her time at Stripe due to the company's rapid growth and reliance on its infrastructure by customers. Uma emphasizes the importance of a blameless culture in incident management, as well as having robust systems in place to handle any issues that arise. She believes that investing time and effort into improving incident management processes can save businesses time and resources in the long run.