September 2014 Summaries

Alexis Lê-Quôc, co-founder and CTO of Datadog, outlines a strategic approach to IT monitoring by classifying metrics into work and resource categories, emphasizing the importance of focusing alerts on actionable work metrics to enhance system effectiveness and reduce unnecessary alerts. Datadog internally tracks a vast array of metrics to convert data into actionable insights, and has honed a method for determining which metrics to monitor and alert on, advocating for regular reviews of metrics and alerts to ensure systems are performing optimally. By prioritizing work metrics, which measure the productivity of systems, and selectively alerting on resource metrics only when they indicate potential failures, teams can preemptively address issues before they impact customers or escalate stress levels. This method aligns with the goals of knowing system performance and minimizing stress, providing a framework for improved monitoring strategies that can be further explored through a trial of Datadog's services.

Testomato provides automated testing solutions to identify issues on websites and forms, emphasizing the importance of understanding and implementing error severity levels to enhance problem resolution efficiency. Severity levels help prioritize issues based on their impact, reducing alert noise and ensuring that the right team members are notified according to the severity of the problem. The blog post discusses the creation of an escalation process tailored to a team's specific needs, which involves structuring severity levels, understanding team organization, and establishing a communication flow. It highlights the necessity of aligning severity structures with team roles and notification channels to ensure problems are addressed promptly and involve only necessary personnel. The authors share insights from Testomato's experience, using tools like PagerDuty for critical alerts and email notifications for lesser issues, demonstrating how structured severity levels and communication paths can optimize issue management and adherence to service level agreements (SLAs).

PagerDuty has rebranded its Advanced Analytics as Advanced Reporting, offering Team, System, and User Reports to give operational insights into people, technology, and processes. This initiative was driven by the need to provide clearer visibility into operations, addressing customer challenges in identifying problematic system areas and team performance. The development focused on reliability and scalability, processing millions of incidents monthly, and employing decoupled services to ensure quick data delivery without affecting core systems. The analytics framework is designed for flexibility and future enhancements, with pre-computed metrics balancing performance and data freshness. Throughout development, customer feedback was integral, leading to improvements through beta testing, which highlighted the system's effectiveness in enhancing operational visibility and performance. Advanced Reporting is initially available as a 30-day preview, after which it becomes part of the Enterprise plan, with developers eager for user feedback to refine the tool further.

New Zealand's position on the Pacific "Ring of Fire" results in frequent seismic activity, with about 10 felt earthquakes daily, and the 2011 Christchurch earthquake highlighting the need for effective response systems. GNS Science, through its GeoNet system, plays a crucial role in monitoring seismic activity with a network of seismometers, although outdated alert systems previously hindered timely notifications to their response teams. The introduction of PagerDuty has revolutionized their alerting process by ensuring that only significant earthquakes, those 4.5 or larger on the Richter scale, trigger alerts to on-call team members, akin to IT incident management practices. This shift not only enhances the team's responsiveness during crises but also allows for flexible management of on-call schedules, ensuring team members can adapt and avoid burnout during extended emergencies. The reliability of PagerDuty in maintaining effective communication during seismic events underscores its importance in public safety, with GNS Science now better equipped to provide timely information to the public and media.

PagerDuty has introduced Advanced Reporting, previously known as Advanced Analytics, as a new feature to help IT Operations teams make sense of the vast amount of data they collect. This feature includes Team, System, and User Reports and is part of PagerDuty Analytics, which provides critical operational insights into people, technology, and processes. The Team Report focuses on metrics like Mean Time to Resolve (MTTR) and Mean Time to Acknowledge (MTTA), allowing teams to track and improve response and resolution times. The System Report helps visualize incident loads and trends, aiding in the reduction of unnecessary alerts and prioritizing engineering tasks. Advanced Reporting is available to all PagerDuty customers during a 30-day public preview, and users are encouraged to explore its capabilities through resources such as webinars and best practices guides.

The blog post discusses strategies to enhance the efficiency of IT operations by improving key performance metrics such as Mean Time to Acknowledge (MTTA) and Mean Time to Respond (MTTR) using PagerDuty. It emphasizes the importance of developing guidelines for acknowledging incidents promptly and advises setting targets for acknowledgment time to ensure teams have control over this metric. The post also highlights the significance of defining clear incident resolution protocols, including automatic resolution through API integrations or manual processes, and the careful use of timeouts to manage acknowledgment and auto-resolution effectively. Additionally, it addresses the issue of flapping alerts, which can skew MTTA and MTTR metrics, suggesting adjustments to monitoring thresholds to mitigate alert fatigue. The post aims to provide best practices for using PagerDuty effectively to maintain operational efficiency and improve response times.

Addressing alert fatigue in IT operations involves utilizing data to refine monitoring systems and eliminate non-actionable alerts, as outlined in a seven-step process. This process includes committing to action by analyzing alert data and dedicating time to streamline alert workflows, like Etsy's "hack week" initiative. It also emphasizes the importance of cutting non-actionable alerts, adjusting alert thresholds using concepts like sensitivity and specificity, and saving non-severe incidents for non-disruptive times. Consolidating related alerts, providing relevant names and descriptions, and ensuring the right people receive the alerts are also key strategies. Regular reviews are crucial to maintain monitoring hygiene, with companies like Etsy incorporating weekly processes such as "Opsweekly" to prevent alert fatigue from becoming the norm. By setting quantifiable metrics for the on-call experience and taking ownership of monitoring hygiene, teams can significantly reduce alert fatigue's impact on IT operations.