August 2024 Summaries

Last year, incident response tool Catalog was released to help organizations contextualize problems within their organization. The aim was to improve incident response by answering critical questions such as which team owns a service behind the checkout page or who needs to be involved if the primary database fails. Over the past year, Catalog has been improved to become "the contextual engine to drive automation across your entire organization." New features include integrating data from any source system, building quickly with AI, not adding complexity as another source of truth, and no vendor lock-in. These improvements have expanded the list of things that customers can do with Catalog, such as automatically populating incident metadata, attribute and route alerts, simplifying declaring incidents, bringing business context to incidents, exploring insights by team, getting the right people in the room, and more.

At incident.io, they prioritize excellent observability to ensure high availability for their customers. They achieve this by creating a user-focused lens on their internal setup, treating it like a product with great UX. Their system follows the principle of "alerts in, notifications out" and is structured around four key areas: alerts, alert routing, escalations, and notifications. Each area has an overview dashboard that provides a high-level view of the system's health, followed by more specific dashboards for each subsystem. They also use event logs to provide a single, consistently formatted log line for each task, and tracing to visualize what a request spent its time doing. The goal is to make their observability setup feel great, with clear structure and hierarchy, and to achieve this through deliberate investment and buy-in from the team.

We've launched incident.io for Microsoft Teams, a major milestone that brings our response tools to a wider audience, expanding on our existing products Response, Status Pages, and On-call, which have been used by iconic companies like Etsy, Hashicorp, and Netflix. With incident.io in Teams, users can create incidents instantly from within a Team, connect alerting systems, and automatically create dedicated Teams channels for each incident to keep everything in one place. The platform delivers clear process information where needed, allows workflows that reduce workload and save time, is supercharged with AI capabilities, and helps users learn from incidents without manual work. This marks the beginning of significant improvements every week, and we're excited to work with Microsoft Teams customers to improve the product.

The author discusses how their team's reliability was a top priority when releasing `On-call`, a new notification system, as they wanted to ensure it could handle pressure from customers with high expectations. To achieve this, the team introduced a library of Smoke Tests that continually test core journeys of the product in production and staging environments, running every minute. These tests exercise the infrastructure and third-party integrations, alerting the team if any fail consistently. The team also runs these tests on every change to the application, both when pull requests are raised and as the merged change is deployed. This approach allows them to make changes without breaking things. The author shares learnings from building their Smoke Test framework, such as starting with a clean slate, using standard Rails, testing user assumptions, preparing for unexpected issues, and adopting a cultural mindset of continuous improvement.

The Head of Growth Marketing at incident.io, Julia Pearson, discusses her favorite memories and experiences working at the company. She highlights the importance of problem-solving and collaboration in finding a way to solve problems individually and collaboratively. The company's benefits, such as having the last Friday of the month off, are also appreciated by Julia. She enjoys participating in the #gratitude channel on Slack, where team members share appreciation for each other's work. Julia advises candidates to review the company's values and implement them throughout the interview process. She describes the culture at incident.io as pace, ambition, and fun, with a focus on making magic happen for customers. Julia is looking forward to SEV0, the company's first conference, and believes that being part of the Marketing team offers ample opportunities for growth and impact. The company values pace and encourages employees to work collaboratively to solve problems and make a positive impact on customers.

SEV0 is an incident management conference aimed at leaders who want to be at the cutting edge of modern incident management, with a focus on practical learning and collaboration. The event features curated speakers from top companies like Netflix, OpenAI, HashiCorp, Y Combinator, and Slack, sharing their experiences on topics such as building effective processes, training for incidents, and balancing reliability and scalability. The conference is designed to be a day of hands-on learning, with the goal of helping attendees build long-lasting relationships with peers and implement actionable advice in their own work.

We've moved to Omni for internal BI tooling, Explo for customer-facing embedded insights in our product, and Hex for ad-hoc analysis. We've improved our local dev setup by running dbt locally with a simplified manifest system that allows us to quickly update our data models without having to recompile the entire database. Our current data stack is built around Extract & Load, Transform, and Analyze sections, with tools like Fivetran for syncing data, dbt for SQL transformations, CircleCI for CI/CD, Hightouch for synced data, Omni for internal analytics, Explo for customer insights, and Hex for ad-hoc analysis. We've also implemented an on-call system using our own product to track and resolve data pipeline issues in a more efficient manner.

This company, incident.io, was facing a bottleneck due to its growing data ecosystem and the inability of its current data platform to scale. The Data Engineer role I joined brought about significant improvements to the development platform, including enhanced dbt workflow and CI, as well as reduced costs through optimized Docker image management and cost-effective Fivetran BigQuery usage. Additionally, exploring alternative data ingestion tools was conducted, with Google Datastream being ruled out due to high upsert costs, and Fivetran remaining the most cost-effective option. The company's platform costs were also analyzed and reduced by nearly 75% through clustering of destination tables in BigQuery, resulting in a more efficient data management workflow.