July 2023 Summaries

During a recent hackathon, participants experimented with integrating OpenAI's GPT into their product for various purposes such as auto-generating incident summaries and allowing users to use natural language for filters. They encountered challenges like token limits, latency issues, and prompt engineering difficulties. Despite these hurdles, they learned valuable lessons about using AI models and identified areas worth revisiting in the future.

Incident.io leverages its Product Responder role to maintain a fast pace of development and address customer issues quickly. The rotative role is played by engineers who focus on resolving bugs, coordinating with the customer success team, and prioritizing issues. This approach allows for clarity in ticket lifecycle management, centralization of noise through designated leaders, shared responsibility among teams, effective communication with customers, acknowledgment of demand surges, and a focus on developer experience. By maintaining this process, Incident.io ensures high product reliability and supports its users even when things go wrong.

Incident.io maintains quality at pace by using "Polish Parties," meetings held towards the end of a project where feedback is gathered from people who were not involved in building it. These sessions involve employees from various departments, including engineering and product managers, designers, and stakeholders. The purpose of Polish Parties is to gain fresh perspectives on the product and identify minor flaws or user experience hiccups that may have been overlooked by those deeply involved in its development. By involving more people, both within and sometimes outside the team, valuable insights into how to improve the product are obtained. The success of these meetings relies heavily on trust and a blameless culture where feedback is constructive rather than harmful or combative.

Site reliability engineering (SRE) is a discipline that combines software engineering and systems administration to establish a framework emphasizing reliability, scalability, and efficiency for modern engineering teams. SREs manage various responsibilities such as service-level objectives (SLOs), incident management, capacity planning, system design consulting, automation, performance optimization, change management, and disaster recovery planning. The guiding principles of SRE include service-level objectives and agreements, error budgets and policies, automation and tooling, monitoring and incident response, and post-incident reviews and continuous improvement. Benefits of implementing SRE in businesses include enhanced service reliability, improved alignment between development and operations teams, and efficient resource utilization leading to cost reduction. Tips for implementing SRE in an organization involve defining SLOs and SLIs upfront, embracing automation, fostering blameless post-mortems, regularly conducting capacity planning, leveraging performance monitoring, and using tools like incident.io to improve product resilience and deep learning from incidents.

The team at incident.io launched Status Pages, a project that aimed for pixel-perfect polish. They worked closely with their designer throughout the process to ensure every detail was well thought out. Key steps included building alignment, brainstorming, creating lo-fi designs, and collaborating synchronously. They also leveraged customer feedback from design partners to refine the product. Polish Parties were held to gather more eyes on the final designs and make necessary adjustments. The team followed principles such as early and iterative feedback, close collaboration with their designer, not being precious about their work, and making decisions based on shared guidelines/values.

In June 2023 newsletter, incident.io highlights the power of their latest product, Catalog, which decisively won against Thanos in a metaphorical showdown. Catalog is designed to be a map of everything that exists within an organization and helps level up incident response by identifying responsible parties for impacted services. The company also launched Learning Flows, allowing teams to define post-incident processes. Additionally, they shared insights on how their product team uses Catalog and announced the refreshment of their Practical Guide to Incident Management with new sections on incident response insights. Lastly, incident.io encourages readers to explore a demo of their platform.

The Barcelona 2023 Company Offsite was a significant event for incident.io, celebrating growth and success while setting the tone for the future with their first values session. The company has experienced substantial expansion in the US, including opening an NY office and hiring employees across various states. They have also released new products like Status Pages and Catalog to enhance incident management. During the offsite, activities included a paella competition, sightseeing tours, flamenco performances, and a values session aimed at updating their company values. The trip was not only fun but also transformative for the team's growth and future direction.

The incident-io/core application uses a combination of environment variables, config files, and secrets stored in Google Secret Manager to configure the app. Config is divided into non-sensitive, sensitive, and runtime configuration types. Non-sensitive and sensitive config are tracked alongside code and expected to be set to the same value for each instance of an application environment. Runtime configuration differs as it's specific to infrastructure and may vary between instances or roles. Config files are stored in `config/environments/<env>.yml` and parsed into a Config structure defined in the code. The app loads config values via a package singleton, which is loaded on application boot. Google Secret Manager securely stores secret material under a secret name with multiple versions. It can be accessed through the Google Cloud Console or APIs. The security model separates secret material from the app runtime by placing secrets in Secret Manager and accessing them using a Google Service Account associated with the app. This limits the possibility of access to a small window between a breach and when credentials are rotated. Restricting secret access is achieved through Access Levels, Security Perimeters, and Google Access Context Manager. These measures help protect against accidental exposure or malicious attempts to extract secrets.

Incident escalation is an integral part of effective incident management, ensuring that issues are addressed by the right people within a reasonable timeframe. An escalation policy serves as a guide for directing incidents to appropriate teams and individuals. Effective incident escalation involves defining clear paths, setting thresholds for escalation, embracing helpful automation, and learning from past incidents to improve future response strategies. Automated tools like incident.io can streamline the process by facilitating quicker incident creation, enabling automated escalations, and providing a platform for continuous improvement through post-incident analysis.

Service Level Indicators (SLIs) are quantitative measures that evaluate the level of service provided by internal teams or service providers. They help maintain customer satisfaction and operational efficiency. There are six critical metrics for effective incident management: response time, error rate, service availability, system throughput, response latency, and compliance. Monitoring and optimizing these metrics help shape Service Level Agreements (SLAs) and ensure realistic agreements with users. The harmony between SLAs and SLIs leads to a well-run service operation and satisfied customers.

Service Level Agreements (SLAs) and Key Performance Indicators (KPIs) are crucial tools in incident management, helping companies streamline their response processes. SLAs set expectations between service providers and clients, while KPIs measure internal performance against these expectations. While they have distinct purposes, both contribute to operational excellence. SLAs cover broader aspects of service, including timelines, finances, benchmarks, and provider-client relationships. In contrast, KPIs focus on specific metrics such as Customer Satisfaction (CSAT) scores, help desk metrics, retention rates, and marketing engagement. Effective SLAs and KPIs require understanding client needs, clear metrics, regular monitoring, and adaptability to changes. Incident management platforms like incident.io can enhance performance metrics and boost customer satisfaction by reducing downtime through better incident management. By streamlining communication and automating response processes, companies can improve their incident response efficiency and build more resilient products.