July 2020 Summaries

Earlier this month, a router misconfiguration within a widely used service provider caused significant internet disruptions, affecting several well-known SaaS companies, which led PagerDuty to observe a global spike in incidents. In response to the unusual increase in incident volume, PagerDuty initiated a Major Incident Response, leveraging its mobile app to coordinate a remote team of incident commanders, subject-matter experts, and stakeholders via Slack and Zoom from various locations, including San Francisco, Toronto, and Atlanta. Their process emphasized the importance of a distributed work culture, allowing the team to rapidly acknowledge, react, and coordinate responses irrespective of physical location. PagerDuty’s Slack integration served as a central hub for real-time communication and documentation, which aided both immediate response efforts and postmortem analysis to enhance future incident management. This incident underscored the efficacy of PagerDuty's platform and practices in maintaining operational continuity for customers during major disruptions, demonstrating that remote orchestration can effectively mirror in-office incident management.

PagerDuty has announced a series of updates and enhancements aimed at improving incident management and response efficiency for IT teams. Key features include Event Intelligence, which utilizes machine learning to reduce noise and enrich incidents with contextual insights, and Dependency-Aware Related Incidents, which provides real-time insights across services. The platform also introduces Paused Incident Notifications to manage alert noise, a new Visibility Console for a consolidated view of service performance, and My On-Call Shifts and its widget to streamline on-call management. Additionally, new REST APIs and automation integrations with tools like Rundeck enhance self-service extensibility and workflow automation, enabling quicker incident resolution. These enhancements are designed to facilitate better incident response, reduce operational noise, and streamline communication across distributed teams, ultimately improving digital operations and customer experience.

Efficient incident management is crucial to minimizing disruptions in service, and PagerDuty emphasizes the importance of clearly defining incidents as unplanned disruptions that affect customer use, while distinguishing major incidents requiring coordinated team responses. The process involves crafting precise alerts that are actionable and appropriately noisy to ensure all incidents can be identified, without overwhelming operators with non-critical notifications. PagerDuty outlines the need to categorize alerts based on priority, urgency, and severity, which guides the response protocol. When an alert qualifies as an incident, effective communication and structured roles, such as Incident Commander and Scribe, are vital in managing and resolving incidents, ensuring engineers focus solely on resolution while other roles handle documentation and communication. This structured approach not only aids in incident resolution but also streamlines alert systems to improve efficiency and response time, ultimately benefiting the service and its users.

In a virtual PagerDuty Summit EMEA, Vodafone's technology leaders outlined their strategy to achieve immutable telemetry, focusing on empowering development teams to be self-sufficient and ensuring their code reaches production swiftly. Central to this approach is building self-service capabilities and continuous CI/CD infrastructure, which requires immutable infrastructure for reliable observability and alerting. The concept of immutability, explained through a Formula 1 analogy, emphasizes replacing broken components quickly rather than repairing them. Vodafone's Site Reliability Engineering teams are tasked with developing on-demand monitoring solutions, allowing teams to choose specific tools without disrupting others. This setup involves integrating configurations for monitoring tools like Datadog and PagerDuty into a CI pipeline, enabling developers to easily implement necessary monitors. The initiative aligns with SRE principles by reducing toil and enhancing visibility across digital operations, and it aims to streamline processes so developers can focus on innovative tasks while maintaining ownership of their code.

Form3, a UK-based payment-as-a-service platform, aims to eliminate slow payment processes by leveraging its cloud-native infrastructure on AWS to deliver faster payment processing compared to traditional, monolithic systems. At PagerDuty’s Summit EMEA 2020, Form3's COO, Eimear O’Connor, highlighted how the company uses PagerDuty alongside AWS to enhance their operational efficiency, ensuring seamless payment experiences for end customers and maximizing revenue capture for merchants. The company, which serves banks, e-commerce platforms, and fintechs, has adopted a DevOps approach fostering accountability and rapid incident response, utilizing PagerDuty for real-time alerting and incident management. As Form3 grows, it incorporates automation into its infrastructure, using tools like PagerDuty’s Terraform integration for streamlined processes, and extending PagerDuty's functionalities beyond engineering to facilitate tasks like tracking on-call hours for financial reporting.

COVID-19 not only poses a direct threat to health and life but also leads to long-term health consequences due to disrupted and delayed healthcare services, disproportionately affecting underrepresented and underresourced communities. The pandemic has exacerbated challenges such as an exponential rise in patients needing acute care, inequities based on race and geography, service disruptions, and the necessity for rapid digital transformation in healthcare systems. In response, PagerDuty.org has launched a fund aimed at improving time-critical health outcomes by supporting organizations that prioritize reaching underserved populations quickly through technology-driven solutions. This initiative offers unrestricted grant funding, including free product licenses and support, to eligible nonprofits, NGOs, social welfare organizations, government entities, and social enterprises, with a focus on projects like vaccine delivery, health hotline reliability, and faster access to care. Applications for the grants are open until July 31, 2020, with grantees to be announced by September 15, 2020, and interested organizations are encouraged to participate in a live Q&A session to learn more about the grant process and PagerDuty's real-time operations platform.

J. Paul Reed, a Senior Applied Resilience Engineer at Netflix, shared insights on postmortem best practices during an Ask Me Anything (AMA) session hosted by PagerDuty, emphasizing the importance of conducting blameless postmortems to foster a culture of trust and continuous improvement. Reed highlighted the need for a blame-aware approach, which involves recognizing biases that might impede objective analysis of incidents, and stressed the role of managers in maintaining a blameless environment while correcting biases. He advocated for maintaining existing systems to build "tribal knowledge" rather than opting for complete replacements, as this enhances problem-solving capabilities during incidents. Furthermore, Reed recommended setting realistic follow-up tasks post-mortem and conducting these reviews within 72 hours of an incident to minimize cognitive biases and ensure data reliability. These practices, according to Reed, not only improve workflows but also contribute to a productive and understanding organizational culture.

PagerDuty has introduced a new integration for Salesforce Service Cloud aimed at enhancing the collaboration between customer service, engineering, and IT teams to resolve customer issues more proactively. This integration allows customer support agents to act as virtual incident commanders, orchestrating responses and actions to various customer-related issues directly within Salesforce Service Cloud. By enabling real-time communication and issue escalation, it helps in providing up-to-date information to customers, thus improving customer satisfaction and retention. The integration supports the synchronization of notes and incident management within the Salesforce interface, allowing support teams to notify the appropriate on-call team about relevant activities and ensuring seamless bi-directional communication. Ultimately, the integration is designed to prevent issues before they begin and to facilitate faster resolution times, thereby meeting the high expectations of modern customers and reducing the risk of losing them due to unsatisfactory service experiences.

PagerDuty's inaugural virtual Summit EMEA highlighted the company's commitment to diversity, equality, and leveraging digital transformation in the face of the COVID-19 pandemic. CEO Jennifer Tejada emphasized the importance of adapting to a completely digital customer experience, building trust through digital channels, and accelerating digital initiatives with cost-cutting efficiencies. She advocated for the implementation of DevOps to foster collaboration and rapid innovation. The event also featured a fireside chat with John Porter, CEO of Telenet, who discussed the cultural and operational shifts towards agility within his organization, emphasizing trust in employees and fostering deeper customer relationships. Porter shared how Telenet is investing in community initiatives like BeCode to support diversity and provide opportunities for minority communities, highlighting the belief that diverse companies perform better. The summit offered additional sessions, including product keynotes and panels on operational efficiency, available for on-demand viewing.

PagerDuty's UX Design team is dedicated to creating user experiences that effectively manage the stress and challenges associated with major incident response, acknowledging that while customers appreciate the service, they dread the alerts signaling trouble. The team, comprised of passionate designers and researchers, focuses on enhancing customer preparedness for incidents by designing intuitive interfaces that cater to varied user needs, from small businesses to large enterprises. They prioritize psychological safety and creativity, especially during the COVID-19 pandemic, fostering an environment where team members feel comfortable experimenting and asking questions. This includes daily check-ins, Zoom challenges, and transforming meetings into game sessions to maintain morale. The team strives to humanize chaos by providing timely signals and developing integrations that empower users to take prompt action, ensuring a seamless incident management process.