October 2022 Summaries

This month at incident.io has been busy with new features and updates. The company launched its new podcast, incident.fm, which discusses various aspects of incident management. They also introduced expressions for workflows, allowing users to vary the behavior of steps within their workflow based on conditions. Additionally, they revamped their follow-ups dashboard with performance metrics and a new incident closure wizard to help teams complete post-incident activities. Lastly, Lisa, the Technical Lead, shared insights into building a great start-up developer experience through short feedback loops, efficient bug tracking, and using well-tested abstractions for new features.

The author shares their experience of joining a small engineering team at incident.io as the seventh Product Engineer. They discuss the unique aspects of working in a start-up, such as pace, impact, and vibes. Pace refers to the high default rate of output and quick adaptation to changes. Impact is about having a significant stake in company decisions and constantly learning from diverse challenges. Vibes involve working with an enthusiastic team that shares a common mission. The author also provides advice on how to choose the right start-up, including attending casual meets, assembling questions, participating in culture interviews, and trusting one's gut feeling.

In this episode, Chris Baynham-Hughes and Pete Yandell from incident.io discuss on-call management best practices. They emphasize the importance of creating a healthy on-call culture by addressing issues such as rotating shifts fairly, ensuring adequate rest between shifts, and providing compensation for on-call responsibilities. The conversation also covers practical tips for operationalizing an on-call team, including effective onboarding processes, encouraging incident declaration and escalation, and conducting game days to simulate real-life scenarios. Additionally, they discuss the benefits of using technology tools like Slack or Microsoft Teams to facilitate communication during incidents. Overall, this episode provides valuable insights into how organizations can improve their on-call management practices to enhance team productivity and reduce burnout among employees.

Incident.fm, a new podcast by incident.io, is set to launch on YouTube and podcast apps later this week. The podcast will feature interviews with industry experts discussing various aspects of incident management. In the first episode, Viktor Stanchev from Anchorage Digital shares insights on running better incidents. Dennis Henry from Okta discusses why "how" is a more appropriate question than "why" after an incident in another episode. Lastly, Colette Alexander, Director of Engineering at HashiCorp, dispels common myths around incident response. To stay updated with the latest news and insights on incident management, sign up for incident.io's monthly newsletter.

Incident.io's Decision Flows feature allows teams to create a series of questions that lead to decisions based on the answers provided. This tool is designed to guide users through critical incidents and help them make informed decisions during crises. Some common uses for Decision Flows include determining incident severity, checking if an issue is with the team's service or a third-party, deciding when to update external status pages, and guiding engineers on how to resolve specific issues. The feature can also be used outside of engineering contexts, such as managing lost or stolen company devices. Decision Flows are flexible and can be tailored to suit various incident management needs.

The article discusses the use of generics and code generation in Go programming language. It explains that generic programming allows writing algorithms with unknown parameters or return types, which can be instantiated when needed for specific types provided as parameters. The author shares their experience at incident.io where they combined generics with code generation to provide useful developer flows and solve some tricky pain points in their codebase. They introduced the concept of partial structs using generics, which represents a subset of a whole struct type and tracks which struct fields it sets. This approach helps in performing safe, partial updates in database records. The author also talks about code generation helpers around Partial types, such as builders and matchers, to improve the experience of using Partials. Overall, the combination of generics and code generation provides a balance between developer experience and runtime safety.

ITSM (IT Service Management) is an umbrella term for all the activities an IT team does to manage and improve the end-to-end delivery of its IT services. ITIL, on the other hand, is a widely accepted set of best practices to deliver ITSM. The ITIL certification provides guidelines on how to implement ITSM activities to the highest possible standard. Incident management is a part of ITIL's "practices" and it involves logging, recording, and resolving unplanned interruptions that disrupt normal delivery of a service. Its goal is to reach incident resolution as quickly as possible so you can resume normal service operation and minimise any negative impact on your business. ITIL 4 doesn't define a prescriptive incident management process but provides some core principles for incident management practice activities, including designing an incident management practice that reacts differently to different incident types based on the impact, prioritising incidents, agreeing classifications and timescales to response with customers up front, and using an incident management tool for logging and managing incidents. Incident management differs from Problem management as it refers to finding the root cause of one or more incidents after they have been resolved and usual operations have been restored. The purpose of problem management is to reduce the likelihood and impact of incidents by identifying actual and potential causes of incidents and managing workarounds and known errors. Incident.io helps teams meet ITIL best practice standards for an incident management process, providing features such as response foundations, incident types, escalation, and communications. It also facilitates the integration of Incident Management and Problem Management by capturing all incident activities in a dashboard to help identify recurring pain points and prioritize areas for more permanent solutions.

The text discusses a guide to creating an incident management software stack, focusing on key activities that businesses need to consider such as collaboration, escalations, customer communication, tracking follow-up work, and documenting processes. It mentions the most common stack used by organizations today, which includes Slack for collaboration, PagerDuty for escalations, Statuspage.io for customer comms, Jira for follow-up work, and Google docs for writing incident debriefs. The text also highlights the pain points faced by those using this combination of products and how incident.io can bring them together in a more powerful way.

The text discusses how tech companies use on-call rotas and alerting tools to proactively identify issues and communicate with customers for better experiences and trust. It explains the setup of incident triggers feature, which creates an incident in a triage state when a PagerDuty alert is triggered. This integration includes Sentry for error reporting and PagerDuty for alerting. The author shares how this feature works, its benefits such as providing one place to go for all related information, filtering out noise, streamlining communication, and avoiding repeated failures. They also mention their experience using the product themselves and encourage others to try it out or reach out for more information.

Incident.io uses a suite of tools for acquiring and retaining customers, focusing on providing a smooth and personalized experience from initial contact through signing contracts. The sales-serve process involves several additional tools such as Sales CRM (Close), scheduling (Calendly), market intelligence (Clearbit), billing (Stripe), contracts (Juro), security (Vanta, Skypher), and gluing everything together (Retool, Zapier). These tools help manage sales interactions, scheduling with customers, quoting customers for services, signing contracts with customers, proving credentials to customers, and retaining customers. Vitally is used as the backbone of their Customer Success team, while Intercom, Loom, Productboard, and Retool support customer queries and provide a control center for product configuration. Zapier connects various systems and synchronizes notes between Sales and Customer Success teams.

The article argues against overly detailed incident response plans, suggesting that they may not be as useful as they seem. It points out that every organization is different, incidents often require input from various teams, and the practicalities of running an incident can be overwhelming. Instead, it recommends encoding and automating processes, normalizing incident declaration across all departments, and practicing incident response regularly to ensure preparedness for critical incidents.

At incident.io, building a great developer experience is crucial for maintaining pace and delivering value to customers. Key strategies include shortening feedback loops through hot-reloading setups, speedy CI, quick code reviews, and Slack previews. Additionally, using strongly-typed languages like Go and TypeScript helps catch errors early on. Building reliable abstractions as reusable building blocks also accelerates feature development. Lastly, making bug-finding easy by leveraging realistic dev environments, traces, sourcemaps, and structured data in logs and errors contributes to faster issue resolution.

At incident.io, a small and fast-moving team, a typical day involves various tasks such as code changes, data team weekly planning, addressing CI pipeline failures, discussing data hiring, product launch check-ins, insights of the week, Donut coffee catch ups, lunch breaks, ad hoc product metrics requests, analytics work, team time meetings, and after-work activities. The team focuses on building a culture of self-served analytics and ensuring that everyone contributes to the data analysis process.

On September 21st, incident.io's talent team attended the Tempo Super Startups Awards held in London. The event recognized startups and scaleups for creating rewarding career opportunities. Incident.io was nominated in four categories: Top 100, Talent Team, Super Recruiter (Head of Talent Dani Unsworth), and Rising Superstar (Esther Delignat). They were placed at #72 on the Top 100 list, marking their first year as a company. The team plans to continue improving in talent management and will return next year for the awards show.