October 2016 Summaries

PagerDuty is responding to a recent outage by addressing two primary issues: the failover approach to DNS problems and the quality of monitoring for the end-to-end customer experience. The company plans to redesign its DNS architecture to implement a multi-master approach utilizing multiple DNS providers, audit DNS TTLs for consistency across its website, APIs, and mobile applications, and develop a runbook for DNS cache flushing. Additionally, PagerDuty aims to enhance real user monitoring with a global perspective and improve the prioritization of resolution steps during disruptions, focusing on critical services. The company also intends to refine its multi-team response process to ensure effective problem-solving by on-call teams. These actions are part of PagerDuty's commitment to enhancing the reliability and availability of its services to meet customer expectations.

On October 21, 2016, PagerDuty experienced an outage and subsequent service degradation due to issues with its primary DNS provider, impacting both external customer access and internal server communication. The company utilizes DNS primarily for routing customer access and internal server communication, and during the outage, engineers quickly identified and confirmed lookup failures, leading to a service disruption for a subset of customers. To mitigate the issue, engineers transitioned from the primary to a secondary DNS provider and manually updated internal server configurations, gradually restoring services over a few hours. Following the recovery, engineers addressed redundant notifications and worked to clear a backlog of events, while committing to releasing a follow-up outlining steps to enhance DNS infrastructure resilience to prevent similar occurrences in the future.

On Friday, October 21, PagerDuty experienced a significant outage caused by sophisticated DDoS attacks on a major DNS provider, resulting in nearly three hours of service disruption, which included 30 minutes of complete unavailability. This incident has led to a backlog of notifications and further DNS-related issues, prompting PagerDuty to conduct a thorough post-mortem analysis. The company acknowledges its failure to meet high expectations and expresses regret, committing to transparency with upcoming posts detailing the event timeline and a root-cause resolution action plan. Despite the challenges of this unforeseen "black swan" event, PagerDuty emphasizes its dedication to learning from the incident to enhance its resilience and reliability, pledging to maintain open communication and improve its services for the future.

PagerDuty has enhanced its email notification system by introducing rich HTML emails, aiming to make notifications more informative and user-friendly. These HTML emails include features such as embedded links to incidents, services, and escalation policies, color-coding for incident status, and inline images with graphs and charts, all designed to facilitate quicker incident triage and response. Additionally, Gmail and Inbox add-ons streamline access to PagerDuty directly from email clients, while full mobile support ensures accessibility on the go. Customers can opt into this feature without extra cost by enabling HTML emails in their profile settings, with the goal of improving the overall notification experience and reducing response times.

Time to Resolution (TTR), or Mean Time to Resolution (MTTR), is a key metric in customer service that measures the average time taken to resolve a customer issue from the moment it is reported until resolution. The goal of minimizing TTR is to enhance customer satisfaction by reducing downtime and ensuring efficient service restoration. However, MTTR can be misleading if not properly contextualized, as it may not account for variations in incident types or the influence of outliers. To address this, organizations can adopt strategies such as categorizing incidents for more targeted MTTR calculations, measuring the percentage of incidents resolved within a target time, and considering the total number of incidents and their cumulative time. Effective reduction of incident resolution time involves implementing a fast and accurate incident management system, reducing alert noise, ensuring quick incident acknowledgment, setting clear priorities, fostering real-time collaboration, and establishing structured response teams with defined roles. By following these practices, organizations can optimize their incident management processes and improve overall service quality.

Organizations are accelerating their digital transformation efforts to remain competitive and deliver immediate value, prompting DevOps and ITOps teams to modernize their software architectures and operational environments. The PagerDuty Virtual Summit, held from November 2-4, 2016, offers a free online platform for industry experts to share strategies and insights on overcoming digital transformation challenges. With 17 sessions featuring speakers and panelists from leading organizations, the event includes an exclusive presentation by Gene Kim on DevOps and its role in driving digital transformation. Attendees can register for any or all sessions to enhance their ITOps and DevOps strategies with actionable insights.

Monitoring is essential for maintaining proactive IT operations, yet the proliferation of monitoring tools has led to a paradox where increased visibility can generate excessive noise, complicating issue resolution. Effective monitoring involves various tools, including Application Performance Monitoring, Log Analysis, Exception Monitoring, Artifact Monitoring, and Incident Management, all of which contribute to a comprehensive view of IT infrastructure but require careful configuration to avoid overwhelming noise. Key performance indicators such as CPU, disk, memory, network utilization, uptime, system events, and security are critical for alerting IT teams to potential issues, and tools like Splunk, Rollbar, and Threat Stack offer cross-platform insights and security intelligence. The goal is to reduce alert noise and shorten the mean time to resolution (MTTR) by establishing a solid monitoring foundation and leveraging incident management platforms like PagerDuty, which aggregate alerts and facilitate swift response actions. By focusing on actionable alerts and integrating incident management, IT operations can achieve enhanced operational agility and prevent the pitfalls of excessive noise that can lead to productivity and financial losses.

PagerDuty's active participation at the 7th annual Splunk.conf event in Orlando highlighted its strong integration and partnership with Splunk, a leading machine data analytics tool. PagerDuty not only integrates with Splunk through native webhooks but also utilizes it internally to enhance its data analysis capabilities. The integration allows incident responders to quickly act on alerts, reducing alert noise and preventing information overload by grouping incident data efficiently. This collaboration was showcased through discussions and demonstrations of best practices at the event, emphasizing the importance of real-time connectivity between people and tools. PagerDuty's presence at the event provided an opportunity to engage with existing and potential clients, gather feedback, and demonstrate new features within the integration, all while reinforcing its commitment to improving the user experience and maintaining strong industry partnerships.

Large, complex systems, particularly in the airline industry, are highly susceptible to failures due to accumulated technical debt from intricate architectures and the fast-paced evolution of digital demands. Airlines, grappling with the interconnectedness and complexity of their IT systems, face significant financial losses and customer dissatisfaction when outages occur, as seen with recent incidents at Southwest, Delta, and British Airways. Addressing these challenges requires a modern approach to incident management, emphasizing the importance of efficient communication, up-to-date monitoring practices, and investment in sophisticated solutions like PagerDuty. The text stresses the necessity for airlines to adopt digital-first strategies and robust incident management processes to minimize downtime and maintain operational efficiency in an increasingly demanding digital landscape.

PagerDuty's presence at Microsoft Ignite highlighted its growing integration with the Microsoft ecosystem, particularly through partnerships with Azure, SharePoint, Yammer, and Power BI teams. Demonstrated integrations included those with Azure Alerts, Operations Management Suite, Systems Center Operations Manager, and Visual Studio Team Foundation Server, showcasing enhanced agile incident management workflows. PagerDuty's capabilities were further emphasized through use cases involving SQL server monitoring and custom on-premises tools. The collaboration is set to deepen, as PagerDuty plans to integrate with Azure's upcoming diagnostics dashboards and is expanding its offerings with Microsoft Flow and PowerApps, enabling users to automate workflows through custom applications. PagerDuty's role within Microsoft's framework aligns with the objective of empowering users and organizations to address business challenges more efficiently, underscoring the value of its integration within the broader Microsoft strategy.