May 2023 Summaries

In the exploration of capturing application state and using ephemeral containers for debugging Kubernetes, the text outlines the importance of deploying specific tools for diagnostics to balance service restoration speed with comprehensive data collection for lasting resolutions. It emphasizes the benefits of capturing both application and environment states to streamline responses and problem-solving, especially in cloud-native platforms like Kubernetes, where containers may lack built-in debugging tools. The discussion also touches on hybrid scenarios involving both containerized and traditional applications, where capturing real-time evidence is crucial for root-cause analysis. Various use cases are provided, such as database debug using SQL RUN Step in Process Automation, application debug with tools like py-spy for Python applications, and traditional infrastructure debug using bash commands and bpftrace. These examples highlight how PagerDuty Process Automation's flexible workflows can be customized to gather and store essential diagnostics, enhancing visibility and decision-making for responders, developers, and operations teams.

PagerDuty has introduced a new method of API access, transitioning from API Keys to a more secure and flexible OAuth2.0 token-based system that allows for granular access control through API scopes. Each object in the PagerDuty REST API now features defined scopes, such as read and write, enabling users to tailor application access precisely to their needs and adhere to the principle of least privilege. This new system involves setting up access via an app, with administrators and account owners managing app creation and token provisioning. Tokens, which replace the previously perpetual API keys, now expire every 30 days, necessitating regular rotation and management. They are issued through the PagerDuty identity service using client credentials, and their access scope can be limited based on organizational security requirements. Scoped OAuth is currently in early access and will be fully available for all accounts by the end of May 2023, with the company encouraging feedback and participation in their community forum for further discussions.

PagerDuty's latest updates to its Operations Cloud are designed to streamline incident management for distributed teams by reducing tool sprawl and manual processes, thus enhancing operational efficiency and business acceleration. Key innovations include the introduction of Custom Fields on Incidents, which centralize incident data for quicker resolution, and enhanced templates for automating stakeholder communications, which improve consistency and efficiency. Additionally, new integrations with ServiceNow and Jira Server allow users to access PagerDuty's powerful workflow automation directly from these platforms, further improving the effectiveness of ITSM tools. The platform also expands Incident Workflow actions to automate various steps, such as creating meetings or reassigning incidents, ultimately enabling organizations to transition from reactive to proactive incident management. These capabilities exemplify PagerDuty’s commitment to providing a cohesive platform that empowers teams to automate critical work, thereby mitigating revenue risks and advancing business operations.

PagerDuty is a platform designed to enhance operational efficiency and achieve significant cost savings, demonstrated by its impact on companies like SAP and Brink's. At SAP, PagerDuty's tools facilitated a digital transformation by streamlining incident responses, reducing the number of responders needed for major incidents by 25%, and improving communication across teams, ultimately reducing response and resolution times. Brink's utilized PagerDuty's automation capabilities to upgrade their cash management operations, cutting down on manual tasks and saving significant engineering hours, while drastically reducing waiting times for developers from two weeks to three minutes. These examples illustrate PagerDuty's ability to provide immediate financial benefits and long-term operational improvements across various industries, making it an essential tool for companies aiming to reduce costs and accelerate growth.

"The Last of Us" series, inspired by the video game of the same name, portrays a post-apocalyptic world where a fungal outbreak has turned much of humanity into zombies, each type representing different challenges and parallels to incident response in technology. The show stars Pedro Pascal and Bella Ramsey as they navigate a world overrun by zombies, such as the unpredictable "Runners," the persistent "Stalkers," the formidable "Clickers," and the devastating "Bloaters," each symbolizing various types of incidents in a tech environment. The narrative suggests that effective incident management requires identifying early warning signs, leveraging automation for recurrent issues, documenting solutions for complex problems, and executing coordinated responses for major incidents. The piece draws parallels between surviving zombie encounters and managing tech incidents, emphasizing the importance of preparedness, knowledge sharing, and strategic response planning to mitigate disruptions and improve digital operations.

PagerDuty.org, the social impact arm of PagerDuty, is dedicated to fostering an equitable world through its innovative community grantmaking program, supported by a donor-advised fund and the Pledge 1% equity commitment. The program empowers Employee Resource Group leaders and regional community social impact leads by granting them decision-making power over the allocation of funds, encouraging a partner-centered approach and equity in philanthropy. Since its formalization in 2022 with a $60,000 allocation, the program has enabled social impact leaders to distribute grants to 12 organizations globally, including Kids in Tech and Empowr, fostering youth mentorship and education. Collaboration with PagerDuty’s Inclusion, Diversity, and Equity team enhances the program's reach and impact, supporting diverse communities, including veterans through partnerships like Hire Heroes USA. With a focus on transparency and shared learning, PagerDuty plans to continue expanding its tools and training for impact leaders, sharing insights with peer communities such as Pledge 1% and Impact Cloud.

In the context of incident management, the challenge of capturing diagnostic data before applying quick fixes, such as redeploying containers, is significant, especially for companies where performance and uptime are crucial. Engineers often spend excessive time gathering evidence after incidents, as some diagnostic data is only accessible during the container's lifetime, requiring Kubernetes expertise to retrieve it using commands like kubectl exec. To streamline this process and reduce incident-related costs, automation is recommended. PagerDuty's Process Automation offers a templatized runbook that automates the retrieval and storage of diagnostic data, enhancing efficiency during incidents. When debugging utilities are absent from containers, Kubernetes Ephemeral Containers allow users to attach debugging tools without modifying or redeploying pods, although this still necessitates advanced Kubernetes knowledge. A new Kubernetes plugin has been developed to leverage ephemeral containers, facilitating automated diagnostics for containers lacking built-in utilities.