August 2022 Summaries

Incidents can be valuable learning experiences for engineers, providing exposure to new technologies and skills, as well as opportunities to meet colleagues and learn from experienced professionals. A healthy incident culture involves declaring many incidents, encouraging participation at all levels, showing work in progress, and avoiding a hero culture where one individual takes on too much of the response burden. By fostering an environment where everyone can learn from incidents, teams can improve their problem-solving abilities and overall performance.

Customer trust and satisfaction are crucial for any business's success. During incidents, effective communication between Engineering and Customer Support can help maintain customer relationships and even turn negative experiences into positive ones. By providing context, explaining actions taken, setting expectations, guiding customers on what to do, and committing to regular updates, businesses can build loyalty and trust with their customers. Companies like Monzo, Cloudflare, and Slack have demonstrated successful communication strategies during incidents. To streamline this process, a Zendesk integration has been developed to keep engineers, support agents, and customers in sync.

Incident.io, an incident management platform, has partnered with Indent, an on-demand access control provider, to streamline the process of granting permissions during incidents. The integration enables time-bound access to cloud apps and infrastructure for on-call responders, ensuring valid and timely access while meeting compliance requirements. Through Slack commands or web app/chrome extension, users can request escalated on-call access, have it auto-revoked after incident resolution, and maintain accurate logs for compliance audits.

This month at incident.io HQ, the company welcomed new joiners and celebrated employee anniversaries. They released a new Zendesk integration for better customer support during incidents, introduced Policies feature to help companies stay within agreed targets, and provided a summary of the Digital Operational Resilience Act (DORA). Additionally, they discussed incident management strategies in an episode of Slight Reliability podcast. Lastly, they encouraged users to review their platform on G2 for potential swag rewards.

In this update on their data stack, incident.io has made several tweaks and improvements over the past two months. They have migrated Fivetran to a separate BigQuery project to deal with concurrent query limits, set up CircleCI to run the dbt pipeline sequentially, moved from using Google Sheets for GBP:USD FX rates to an API feed, materialized all of their dbt models as tables, introduced intermediate tables into their dbt pipeline, and used SQLFluff VSCode extension for automatic formatting and error detection. They have also updated their folder structure in Metabase to differentiate one-off analyses from those used by stakeholders across the company.

Incident.io is currently recruiting for their Data team, and they have outlined a detailed interview process that includes an initial call, technical call, take-home task, and on-site interviews. The company emphasizes the importance of mindset and technical knowledge in their selection process. They also encourage candidates to ask questions about the company culture and environment during the interviews. The final stage involves a skills interview and a culture fit interview, both aimed at ensuring that the candidate is a good fit for the team and the company as a whole.

The use of Slack channels for incidents has been a topic of discussion, with two main approaches being considered: creating a new channel for each incident or using a central, long-lived channel to respond to an incident. While the former approach provides clarity and easy context loading, it can lead to more channels being created and potentially overwhelm Slack admins. The latter approach keeps all information in one place but may result in a noisy main channel and require stakeholders to join other teams' channels. Both approaches have their merits, and the choice between them depends on an organization's specific needs and preferences.

James, a full-time designer, recently joined incident.io after working as a contractor for a few months. He has over ten years of experience in design and coding, having worked on various roles such as designer, engineer, or a combination of both. James is passionate about combining disciplines to work efficiently and appreciates being part of teams that prioritize collaboration and shipping impactful projects. In his free time, he creates art and holds a degree in Illustration.

The Digital Finance Strategy aims to support digital finance in Europe while maintaining financial stability and consumer protection. Its three main components include the Digital Operational Resilience Act (DORA), Markets in Crypto-assets (MiCA) Regulation, and a proposal on Distributed Ledger Technology (DLT). The proposed DORA is designed to address increasing information and communications technology (ICT) risks in the financial services sector. It will apply to various financial entities such as credit institutions, electronic money institutions, investment firms, insurance undertakings, reinsurance undertakings, and critical ICT third-party service providers. The proposal focuses on improving incident management at financial entities by introducing better ICT risk management and governance, enhancing incident reporting, ensuring strategies for managing ICT third-party risks, and implementing resilience assessments. Financial institutions will be required to establish and apply incident management processes, classify incidents according to shared criteria, report major ICT-related incidents to the competent authority using a harmonized report framework, and involve senior management in incident management. Non-compliance with DORA could result in significant repercussions, including temporary or permanent cessation of practices or conduct deemed necessary by authorities.

Incident.io has introduced Zapier integration, allowing users to automate repetitive tasks between two or more apps without coding. This enables the creation of incidents from various sources such as high severity Zendesk tickets, Intercom critical tickets, new items in Notion databases, escalated JIRA tickets, and even emoji reactions in Slack channels. Additionally, users can automate post-incident administrative tasks like scheduling retrospectives in Google Calendar or adding rows to other department's BigQuery instances. The integration aims to help users focus on fixing incidents rather than managing repetitive processes.

Meg has recently joined the incident.io team as a Talent Acquisition Coordinator, transitioning from Beacon where she worked on digital supply chain and freight platforms. She is enthusiastic about contributing to building the Talent Operations function at incident.io. In her personal life, Meg enjoys spending time with her dog Bella and socializing in London.

Lucy has recently joined the incident.io team as a Customer Success Manager, transitioning from her previous role at Enthuse where she worked in onboarding and support for charitable causes. She is excited to bring her experience to the customer success team and aims to create maximum delight for their customers. In her free time, Lucy enjoys playing tennis and experimenting with complex dishes in the kitchen.

Interrupts in software development can hinder productivity and impede feature delivery. Often, teams tackle this issue by dedicating an entire planning cycle to addressing technical debt. However, incident.io has reframed interrupts as an advantage by setting up their engineering team to cater for unplanned work. They introduced a "Product Responder" role, which rotates among team members weekly. This role involves handling alerts, errors, and requests from other teams and customers while also encouraging knowledge distribution and process iteration. By addressing interrupts proactively, the company can deliver faster tactical changes that benefit customers and improve overall productivity.

Charlie has recently joined incident.io as an Office & Culture Manager, bringing excitement and enthusiasm to their new role. In their free time, they enjoy spending time with family and friends in southeast London, often exploring the coast through kayaking or paddleboarding. Prior to this position, Charlie worked at Stripe where they led workplace operations in London and participated in various office projects. They are eager to contribute to creating a high-quality work environment that aligns with the exceptional team at incident.io.

Connie has joined incident.io as a Customer Success specialist, taking over from Esther. She is excited about working with the company's customers and being part of its growth. Before this role, she worked at Pento where she built and led their customer success team. In her free time, Connie enjoys cooking, swimming, doing yoga, and watching TV series. She also owns one of incident.io's office dogs.

Ed has recently joined incident.io as a product analyst and is the first member of their data team. Prior to this, he worked at Arrival, an EV startup focused on sustainability, where he helped define the GTM plan and product roadmap. Before that, Ed founded and scaled an analytics think-tank to approximately 100 people, working with NGOs like Wikipedia. At incident.io, Ed is excited about being part of a promising early team and looks forward to contributing to building an enduring business while learning and growing. In his personal life, he enjoys spending time with loved ones, traveling, and exploring nature.

Jack has recently joined incident.io as a Product Analyst in their Data team, where he works alongside Ed. He brings experience from analytics and data science, having previously worked with some of the current team members at Monzo. Since joining incident.io, Jack's work has been diverse, covering areas such as Growth, Product, and Finance. The company is building on Lawrence's previous work to enhance their data stack, and they are now making progress in analytics. In his personal life, Jack enjoys boxing, traveling, and watching Formula 1 races.

Rory has recently joined incident.io as a Product Engineer after previously working at Freetrade and Fanduel. He contributed to the growth of Freetrade from 50k users to 1.4 million during his tenure there. Rory is excited about joining incident.io, where he aims to develop a product that improves the incident management experience for companies. In his free time, he enjoys cycling, eating out, and playing guitar.

Effective communication during incident response is crucial in maintaining customer trust and minimizing negative impacts on businesses. A customer-centric approach involves making communication a mandatory part of the response process, involving someone close to customers, not waiting for all information before sending updates, learning from customers' needs, being flexible with communication methods, and ensuring transparency while handling sensitive incidents. Atlassian's recent outage serves as an example where poor communication led to lasting effects on their organization. By incorporating these tactics, businesses can foster trust and strengthen relationships with their customers during challenging times.

The company recently opened an office in New York as part of their $34 million fundraise. The majority of the company's customers are from the US, and having a presence there allows them to better support these clients by increasing available hours for sales and customer service by 50%. Additionally, the US offers access to a deep talent pool that complements the London team. The New York office is not considered a satellite location but rather an integral part of the company's growth strategy.

The text discusses the importance of having an effective post-incident process for learning from mistakes, improving future responses, and building resilience in products and teams. It emphasizes the significance of creating a compelling story through a timeline that captures everyone's thoughts and inputs during an incident. To facilitate this, new features have been introduced to help users build clearer narratives for their incidents, including hiding irrelevant events, adding timeline notes, and enabling collaboration through commenting on the timeline.