July 2024 Summaries

The culture at incident.io is described as fun, fast-paced, and exciting, with a strong emphasis on collaboration and teamwork within the Sales team. The company values its employees' hard work and drive, expecting them to be highly productive and efficient. The team's collaborative approach is evident in their use of Slack channels, including #gratitude, where they celebrate achievements and appreciate each other's contributions. Sofie Vanhal, a Commercial Account Executive at incident.io, shares her enthusiasm for the company's culture and values, highlighting the benefits of working with a fast-paced and driven team. Overall, the incident.io culture prioritizes teamwork, productivity, and employee appreciation, creating a positive and supportive work environment.

We analyzed 14,000 follow-ups across various companies to determine how quickly they complete these tasks after resolving an incident. The median time for completing a follow-up action is seven days, with smaller companies completing them faster and larger companies taking longer. Medium-sized companies tend to be the slowest due to coordination and process issues. Post-incident tasks take one day on average, but some tasks are quicker than others, such as reviewing timestamps and metadata. The most time-consuming task is creating a postmortem document, which takes an average of two hours to complete, with 4 days needed for sharing the final document. Companies that complete their postmortems quickly tend to be faster in general, highlighting the importance of efficient incident management processes.

At incident.io, the Data team has implemented an effective data incident management process to handle growing data-related incidents, leveraging their own product to monitor, triage, and respond to them. They've established a structured process with designated on-call engineers, configuring schedules, alert triggers, and response procedures to ensure prompt attention and clear communication of updates to stakeholders. The team uses their product's features to automate tasks, such as sending cover requests from mobile, and has implemented measures to prevent "noise" in incident reports, focusing instead on false negatives.

Behind the Flame is a series that showcases the incidentios team at incident.io. Rory Bain, a Product Engineer on the On-call team, shares his experiences and insights about working at incident.io. He values the "Find a Way" principle, which encourages innovation and taking initiative. His favorite benefits include the last Friday of the month, where everyone gets to relax together, and the #things-jack-says Slack channel, where people share quotes from Jack Buckmelter. Rory advises candidates interviewing for the role to be themselves and showcase their technical skills. He describes the culture as "fun, fast, and freaky." The On-call team is enthusiastic about working on a product that improves incident management, and they recognize achievements with small ceremonies and Slack channels.

The company, incident.io, presents a weekly data insight called the Tasty Data Morsel, where a member of the Data team shares a simple chart with key findings in five minutes or less during all-hands meetings. This practice aims to keep the meetings a mix of business and fun, encouraging questions about the product and its features, leading to discoveries that can improve the product. The data insights also serve as a chance for the Data team to remind everyone of their capabilities and make the data more visible within the company. By sharing these brief presentations, the team fosters a culture of curiosity, encourages public speaking skills, and brings joy through storytelling about the product's features and successes.

We introduced On-call just a few months ago with a foundational set of features, marking only the beginning of our vision for the product. To address uncertainties and maintain momentum in development and delivery, we dedicated a week to focus on scoping key upcoming projects in our roadmap. The primary goal was to turn broad themes into well-defined project plans that were technically scoped and sized. This approach aimed to help us build a roadmap that we could confidently commit to, ensuring the delivery of projects in a sensible order. We wanted to create space for collaboration, building a roadmap that incorporated product and customer expertise, aligned with long-term goals, and was technically deliverable. The scoping week involved four themes, each with assigned pairs working on proposal documents, engineering day estimates, and clear next steps. The team provided robust challenges during the review phase, ensuring everyone understood the specific project plans and identified potential problems early. The technical scoping focused on in-depth implementation details, while the final call for estimations ensured detailed planning. The resulting roadmap was impactful, with a collaborative effort between product, design, and engineering functions leading to better outcomes and shared perspectives enhancing the planning process.

The recent CrowdStrike update has triggered widespread disruptions across various sectors, causing chaos and affecting multiple aspects of society, including flights, emergency services, payment systems, news channels, and transportation. The impact on those who manage such crises is significant, requiring immediate response, technical remediation, and coordination among cross-functional teams to mitigate further damage and restore services. Effective incident response requires preparedness, integrated planning, training, clear communication, and a culture of readiness to address the growing complexity and potential impact of incidents in our increasingly digital world.

Building an on-call product requires careful consideration of time-related complexities, including local time, time offsets, and timezones. The scheduler must accurately calculate which users are on shift at any given time for a specific schedule configuration, taking into account factors such as DST changes, timezone differences, and user preferences. To achieve this, the system uses Postgres to handle timezone conversion and instants, with the application layer working with instants as much as possible. The UI displays times in local time when necessary, while the scheduler is designed to be flexible and fast, with a robust testing strategy that includes unit testing, snapshot testing, runtime auditing, and architectural considerations.

Behind the Flame is a series showcasing incident.io's interesting stories and experiences from its employees. Georgie Dorling, a Customer Success Manager at incident.io, shares her favorite memories, values, benefits, and insights about working at the company. She highlights the importance of transparency, collaboration, and employee recognition in creating a unique culture that resonates with customers. Georgie also emphasizes the value of Win Together, which represents the collaborative approach to achieving goals, and how it influences her work as a Customer Success Manager. Throughout the conversation, Georgie shares anecdotes about customer interactions, her favorite Slack channels, and what she's looking forward to in the future of incident.io.

Linear is a solution for modern product development, known for its fast and effortless use, as well as its beautifully designed interface. The company has a unique approach to product development, assembling short-lived teams to build new features and requiring every engineer to seek out minor design and UX improvements on a weekly basis. This approach is redefining the playbook for software development. The company's focus on teamwork, continuous improvement, and attention to detail has led to its success in delivering a fantastic product.

We are honored to share our Summer 2024 accolades from G2, including being ranked #1 in G2’s Relationship Index, a recognition that reflects our commitment to building strong relationships with our customers. This ranking is based on factors such as reported ease of doing business with the seller, satisfaction with product quality of support, likelihood to recommend each product, and number of reviews received on G2. Our customers are at the forefront of everything we do, and we take pride in forming connections and celebrating milestones with them. We believe that our customers aren't just deals or numbers but rather people who deserve a human-centric approach to their experience with us. Our Customer Success team works closely with Product and Engineering to ensure maximum success for our customers, and this recognition highlights the hard work and dedication of our entire team. The award is a testament to the trust and collaboration from our incredible customers, and we're grateful to be part of a company that views its customers as an extension of its own.

At incident.io, the company has adopted database migrations as a key aspect of their product growth strategy. They have developed a runbook for safely handling migrations, including a recent focus on migrating JSONB columns in Postgres databases. To achieve this migration, they used interfaces `sql.Scanner` and `driver.Valuer` to provide custom marshaling logic for reading and writing data in both the old and new formats. The company has also implemented an async backfill process using this custom scanner to unify the state of the existing data, resulting in a consistent and up-to-date database schema.

At incident.io, they use infrastructure as code to configure everything from Google Cloud Platform to Spacelift and even their own product, incident.io. New engineers start by making a PR to the infrastructure repo on their first day, which sets up their account, team, and development environment. The process automatically configures resources in Google Cloud Platform based on the user's role and team assignments. Additionally, incident.io uses Terraform for its catalog product, allowing it to be configured via code. The company promotes best practice of configuring all products, including itself, in the same way as other products, making onboarding simple and efficient.

This week, Hank Jacobs, Staff Site Reliability Engineer at Netflix, discusses their deployment of incident.io across the organization, highlighting how great UX has enabled rapid rollouts to hundreds of teams and how this partnership has been a game-changer for Netflix. With more entries in their Catalog than any other customer, Netflix is leveraging incident.io's capabilities to drive reliability across their organization. The company's experience with incident.io has been instrumental in improving their overall incident management processes.