August 2022 Summaries

PagerDuty has announced a series of updates and enhancements to its Operations Cloud, including the introduction of new integrations and improvements to existing products. The updates feature advancements like the PagerDuty Process Automation Version 4.5.0, which now includes new plugins such as the Sumo Logic Job Step Plugin and the AWS ECS Node Executor Plugin, enabling users to automate operational tasks more efficiently. The company's Partner Integrations and App Ecosystem have also been enhanced, with notable updates to the PagerDuty App for Jira and a transfer of ownership for the BMC Helix/Remedy integration to partner KTSL. Additionally, PagerDuty is phasing out its Event Rules in favor of the more robust Event Orchestration, with a planned migration for existing users. There are also important changes regarding webhooks, requiring users to transition to V3 webhook subscriptions. PagerDuty continues to engage its community through various webinars, events, and Twitch streams, providing insights and best practices to optimize automation and incident resolution.

Phylicia Jones, affectionately known as PJ, was a cherished colleague and friend at PagerDuty, whose recent passing has left a profound impact on her community. Renowned for her vibrant personality and relentless dedication, PJ was an influential leader who embodied the company's core values and inspired everyone around her with her energy and commitment to excellence. She played a pivotal role in developing leadership programs and advocated passionately for PagerDuty’s employees and culture, particularly through her involvement with the Array Employee Resource Group. Beyond her professional accomplishments, PJ was deeply committed to causes like the Lupus Foundation of America and Beyond Emancipation. Her legacy is celebrated through the renaming of the company’s annual leadership award in her honor and the upcoming Celebration of Life planned by her family. PJ’s influence extended far beyond her immediate circle, leaving a lasting legacy of courage, authenticity, and a relentless pursuit of excellence, as well as a deep void that her colleagues and loved ones are dedicated to honoring by carrying forward her principles and passion.

PagerDuty Summit 2022 was a global event that highlighted the significance of automation in digital transformation, visiting cities like San Francisco, Sydney, and London, while also engaging thousands of online attendees. The summit featured sessions emphasizing the importance of automation in SRE and DevOps workflows to reduce repetitive tasks and improve efficiency, with new features and enhancements to PagerDuty's automation products such as Process Automation, Runbook Automation, and Rundeck Open Source. Key sessions included Jamie Vernon's insights on gaining stakeholder buy-in for automation, Sean Noble's discussion on PagerDuty Automation Actions for incident mitigation, and Scott McAllister's guidance on using Terraform to manage PagerDuty configurations for improved reliability. The event also addressed infrastructure management through code and the importance of setting and meeting Service Level Objectives (SLOs) to ensure service reliability, with Liz Fong-Jones discussing the sociotechnical implications of failing to meet SLOs. The summit's sessions, featuring contributions from industry leaders like Salesforce, Google, Buildkite, and Metrist, are available on PagerDuty's YouTube channel for further exploration.

Service ownership, a key DevOps practice, is increasingly adopted by companies to enhance customer experience and align development teams with business objectives, though it presents challenges, particularly for large organizations managing numerous services. PagerDuty addresses these challenges with the introduction of Service Standards, which provide a structured framework for configuring services according to best practices, promoting consistency and accountability across teams. This initiative helps teams understand what constitutes a well-configured service, supports managers and administrators in scaling service ownership, and ultimately improves incident response and operational maturity. By offering nine configurable standards, Service Standards enable organizations to standardize service configurations, ensure services are informative and integrated, and facilitate the management of service performance data to drive accountability and show progress.

Organizations are increasingly transitioning to cloud-based systems and implementing DevOps at scale to foster innovation, which has led to more distributed applications. Customers expect seamless service regardless of the complexity behind the scenes, necessitating centralized management of teams, services, and data. Integrating IT service management tools with digital operations platforms, such as combining PagerDuty with ServiceNow, can improve incident response times by automating actions and maintaining comprehensive activity histories. IHS Markit, a company providing analytics to various sectors, exemplifies this approach by integrating PagerDuty and ServiceNow to centralize incident management, thereby accommodating the needs of both DevOps and traditional IT teams while ensuring regulatory compliance and management oversight. This integration allows for agile incident management, rapid response to incidents, and enhanced visibility across the organization, all of which are crucial for IHS Markit as it continues to expand and refine its operational model.

PagerDuty has launched an Early Access program for its enhanced Dedicated Incident Slack Channel, which allows users to view incident details, perform actions, and add responders directly within Slack, as well as set up dedicated Zoom conference bridges. To utilize this feature, users must upgrade to Slack on WebHooks V3 and request Early Access from PagerDuty support. The creation of a dedicated incident channel can be done via Slack or PagerDuty's web interface, with options to select or create new channels that can be public or private. The company plans to release this functionality to all users of Slack on WebHooks V3 next month and is working on further improvements to streamline and automate the channel creation process.

PagerDuty's Early Warning System (EWS) model is a crucial tool used by the Customer Success and Sales departments to evaluate customer wellness and predict potential account churn based on product usage and external business factors. In January 2021, an upstream change led to an inaccurate customer risk score, highlighting the need for robust monitoring to prevent such issues in the future. The DataDuty team implemented automated tests and PagerDuty alerts to detect model drift, which can manifest as concept, data, or upstream drift. The team's strategy involves using various statistical tests to ensure the model's accuracy, such as Cohen’s d, kurtosis, the Kolmogorov–Smirnov test, and T-Tests, while maintaining data integrity through tools like Snowflake, Monte Carlo, Apache Airflow, and Databricks. An incident involving a data anomaly was swiftly resolved using these measures, demonstrating the effectiveness of PagerDuty's incident triage capabilities. This proactive approach ensures model reliability and maintains stakeholder trust, emphasizing the importance of monitoring and alerting systems in managing critical data science infrastructure.

PagerDuty's Root Cause Analysis features are designed to help responders quickly resolve incidents by providing context and actionable intelligence. After acknowledging an incident, responders can investigate the root cause using various tools on the incident details page. These tools include the Outlier Incident classification label, which indicates whether an incident is frequent, rare, or an anomaly, as well as the Past Incidents tab, which shows a heat map and details of similar incidents on the same service. The Related Incidents tab provides information on ongoing incidents across all services, enhancing understanding of the incident's scope. Additionally, the Probable Origins widget calculates likely origins based on historical data, and the Change Correlation section highlights recent infrastructure or code changes that might be relevant. These features, combined with Noise Reduction and Event Orchestration capabilities, aim to expedite incident resolution and minimize downtime. PagerDuty emphasizes that these tools create context for faster resolution rather than predicting a definitive root cause, aligning with industry trends to focus on probable or proximate causes.

PagerDuty has been recognized as a Leader in the 2022 GigaOm Radar for AIOps Solutions, noted for its ability to integrate seamlessly into existing IT infrastructures without the need for significant tool displacement. The company’s SaaS platform is praised for its ease of implementation, offering more than 650 integrations and a simplified setup that provides quick time to value while minimizing the need for customization. Notable features include an intuitive user interface, Terraform and API providers, and capabilities in event orchestration that allow teams to efficiently manage and respond to incidents. Automation plays a key role in their offering, with built-in Automation Actions and flexible workflows that balance workloads between human and machine responders, enhancing productivity and reducing burnout. Despite this being their inaugural appearance in the Radar report, PagerDuty has built on past successes with Event Intelligence and is processing over 20 billion events annually, leveraging machine learning and analytics to improve problem resolution and noise reduction.

Modern cloud architectures, especially those centered on AWS, involve a complex integration of around 250 AWS services and numerous SaaS, in-house, and legacy systems, making incident response challenging due to the need for specialized expertise. To address this complexity, new AWS plugins for Automated Diagnostics have been introduced, enabling users to streamline diagnostics across AWS environments by providing tools such as the CloudWatch Logs, Systems Manager, ECS Remote Command, and Lambda Custom Code Workflow plugins. These plugins facilitate automated diagnostics, improving efficiency in configuration management, real-time data retrieval, and custom script execution. Additionally, pre-built job templates are available to expedite incident response processes, offering users a curated set of job definitions to enhance incident investigation and triage. The initiative aims to simplify the integration of AWS diagnostics into existing systems, thereby enhancing service quality and response effectiveness.

Event Orchestration, introduced by PagerDuty, is an advanced progression from the previous Event Rules feature, designed to streamline event management by reducing manual work and automating processes. It uses the newly developed PagerDuty Condition Language (PCL) to handle complex event processing and provides features like improved UI, advanced conditions, and support for APIs and Terraform. This tool enables users to compress rule volumes, enhance noise reduction, and automate routine tasks, offering functionalities like suppression, routing and enrichment, runbook provision, and automated diagnostics. Event Orchestration supports precise event processing with rule nesting and can trigger webhooks or automation actions, making it significantly more efficient than its predecessor. PagerDuty plans to phase out Event Rules and migrate all customers to Event Orchestration to focus on delivering a robust event-driven enrichment and automation experience.

Recent years have heightened customer demands, causing 68% of customer service agents to feel overwhelmed, according to a Zendesk CX Trends report. PagerDuty believes that enhancing the workflow of these agents leads to improved customer interactions and stronger brand relationships. To this end, PagerDuty has deepened its integration with Zendesk, introducing PagerDuty Automation Actions within the PagerDuty application for Zendesk. This feature enables agents to quickly diagnose and resolve incidents by connecting to corrective automation, which reduces resolution times and alleviates the workload on backend teams. Automation Actions empower agents to handle customer-impacting issues efficiently, reducing the need for escalation to engineering teams and allowing them to focus more on customer relationship management. Automation not only standardizes responses to recurring problems but also minimizes mistakes in manual tasks, ultimately improving agents' efficiency and reducing time to resolution.