November 2022 Summaries

On November 18th, 2022, incident.io experienced an intermittent downtime of 13 minutes from 15:40 to 16:12 GMT due to repeated crashes. The team quickly responded by investigating the cause and implementing measures to improve reliability. They discovered that a bad Pub/Sub event was causing the app to crash, leading them to purge several subscriptions and temporarily turn off non-critical parts of the app. They also added panic handlers to ensure all goroutines are covered and physically split their application by work type for increased reliability and performance.

This month's highlights at incident.io include improvements to incident insights with four new dashboards for workload, MTTX, readiness, and seasonality tracking. Additionally, duration metrics can now be configured on the homepage. A new feature called incident subscriptions allows users to receive notifications about changes to incidents via Slack or email. Lastly, a podcast was released discussing foundations of building an incident response process. Users are encouraged to review incident.io on G2 for exclusive swag rewards.

Eva has recently joined incident.io as Head of Legal after previously working at Tessian and Stuart Delivery. She is excited to be part of incident.io's growth phase and looks forward to working on various projects. In her personal life, Eva enjoys walking her dog, long-distance running, reading, attending the theatre, and discovering new dining experiences.

Maddie has joined incident.io as their first full-time People Operations Lead after spending 18 months in a similar role at Freetrade, where she onboarded over 200 employees. She is excited to be part of an early-stage company and believes they are on the verge of something amazing. Outside work, Maddie enjoys good food, particularly brunch, and stays active with netball, tennis, and running, including training for the London Marathon.

In their first month at incident.io, the author experienced several surprising aspects of the company culture and work environment. These included a fast-paced work style with quick feature delivery, the value of in-person collaboration, immediate trust from colleagues, an excellent developer experience, and the opportunity to use the company's product daily. The author expressed excitement for future growth and learning opportunities at incident.io.

This blog discusses the challenges faced by early-stage data teams with a focus on balancing various aspects of their work. The author highlights five core areas of work for data teams: infrastructure, core dashboards, measuring success, deep dives, and ad-hoc tasks. They provide insights into what good and bad looks like in each area and offer tips on how to strike the right balance. The blog emphasizes the importance of trust in data infrastructure, regular check-ins for core dashboards, a consistent framework for measuring success, taking time for deep dives, and managing ad-hoc tasks effectively.

In this episode, Pete and Chris discuss how incident management has evolved over time and how it varies across different organizations. They emphasize the importance of having a clear definition of an incident and share their thoughts on severities and statuses. Additionally, they explore the role of data in incident management and provide examples of non-engineering incidents that have occurred within their own careers. Some key takeaways from this episode include: 1. The concept of an "incident" has evolved over time, with more organizations embracing a broader definition that includes minor issues or bugs. This shift in perspective can help reduce the fear and stigma associated with incidents. 2. Severities and statuses play a crucial role in incident management, as they provide a framework for understanding and categorizing different types of incidents. While it's important to have some level of standardization across an organization, teams should also be flexible in adapting these models to their specific needs. 3. Data is essential for effective incident management, both during the live response phase and after the fact when analyzing trends and patterns. Structured data can help organizations make more informed decisions about risk mitigation and resource allocation. 4. While many incidents tend to be technical in nature, non-engineering incidents can also occur within an organization. These types of incidents may require different approaches or skill sets but still benefit from a structured incident management process.

Alicia has recently joined incident.io as their first Engineering Manager, previously holding a similar role at LabGenius where she worked with the lab automation and DNA sequencing teams. She is looking forward to helping the team at incident.io navigate growing pains while building innovative solutions. In her free time, Alicia enjoys gaming, reading, and taking care of her cats.

Ashley is the new Senior Talent Partner at incident.io, focusing on Tech & Product roles. With over a decade of experience as a technical recruiter, she has worked with small agencies and helped scale companies like Checkout.com from 300 to 2,000 employees. Her goal at incident.io is to help the company grow its engineering and product teams while improving the overall talent acquisition experience. In her free time, Ashley enjoys playing football, watching sports, cooking shows, and spending time with her Staffy named Darcy.

Benji has recently joined incident.io as a Product Engineer after completing an MSc in Machine Learning. Previously, he worked as a software developer at Orgvue where he developed a user configurable data ingest pipeline and a system for anonymising and aggregating product usage data. He enjoys clear communication of complex information and is excited to work on a product that focuses on this during stressful situations. In his first week, Benji was impressed with the team's dedication to addressing users' pain points quickly. When not working, he practices martial arts, plays board games, and writes.

Charlie is a new engineer joining the incident.io team after working at Vidsy, an AdTech startup where he contributed in various roles such as building out the platform, talking to partners, scaling the team from 4 to 120 employees, and fixing office WiFi. Before that, Charlie worked at BBC News, developing systems for election product experiences. He enjoys the fast pace of product feature development at incident.io and looks forward to continuing communication with customers to quickly ship projects. In his free time, Charlie is passionate about coffee, photography, training at The Arch in Bermondsey, and preparing for a climbing trip next Spring to Magic Wood.

Jack has recently joined incident.io as their first US hire after moving from Denver, Colorado to New York City. He is currently in London for his first 5 weeks of onboarding and meeting the team. Despite feeling like a daunting task initially, he quickly felt at home with the company. As the first American employee, Jack has experienced British culture and learned new terms such as "dungarees" for overalls. He is excited to see what the future holds as they grow and continue to work together.

Nick has recently joined incident.io as an Account Executive in New York after working at Frame.io, a video collaboration platform that was acquired by Adobe. Prior to that, he worked at Handshake, a B2B e-commerce start up acquired by Shopify. Despite not being an engineer, Nick believes that managing platform incidents is crucial for maintaining good client relationships and product evaluations. In his free time, he enjoys golfing and exploring new restaurants in the city.

Sam has recently joined incident.io as a Product Engineer after working at Monzo where he was the tech lead for user growth and referrals scheme. He played a significant role in launching a new business account and managing onboarding processes. Prior to that, Sam worked at SoundCloud in Berlin and experienced paging during late-night Drake uploads. In his free time, he enjoys electronic music, cooking, and making unofficial merchandise for London startups. He is excited about contributing to projects, meeting customers, and enhancing incident management tools.

Travers has recently joined incident.io as a founding member of their US GTM team, after previously working in investment research, advising at Merrill Lynch, and on the sales team at Airkit. Currently based in London for his first five weeks of onboarding, he is impressed by the quality of the team and the work being done. Travers plans to establish a presence in NYC soon. In his free time, he enjoys playing tennis, watching matches, and exploring with his corgi Baloo and partner Meredith.