December 2022 Summaries

In 2023, the focus is on reducing routine manual tasks in engineering through increased automation to allow more time for innovation. The blog highlights the impact of toil, as evidenced by a PagerDuty survey indicating that 43% of respondents spend a significant portion of their time on repetitive work. To address this, organizations are encouraged to automate tasks such as server restarts and infrastructure provisioning using tools like PagerDuty Process Automation or Rundeck. The blog emphasizes the importance of delegating automated processes as self-service to end users and suggests reducing interruptions by automating diagnostics for recurring incidents. It also discusses the benefits of integrating existing automation tools with platforms like Ansible to enhance workflow orchestration and security. The blog encourages sharing New Year's resolutions for automation and offers resources to learn more about implementing these strategies.

As the holiday season approaches, many engineering organizations face the dilemma of implementing deploy freezes to prevent bugs during high-traffic periods, while also dealing with reduced staff availability due to vacations. The nuanced debate around deploy freezes highlights that while they aim to reduce incidents, they do not guarantee an incident-free period, as issues can still arise from factors like sudden traffic changes or expired certificates. The blog emphasizes the importance of managing expectations and communicating changes to prevent disruptions and ensure smooth operations for both customers and employees. It also advocates for fair treatment of on-call staff, suggesting mandatory additional time off and holiday pay for those working during holidays. The post encourages organizations to use the holiday period for non-deployable tasks and to conduct retrospectives to improve future deploy strategies, recognizing that incidents are opportunities for learning and improvement in team dynamics.

IT operations face challenges of excessive workloads, interruptions, and pressure from executives, which have led organizations to rely heavily on ticket-driven work management systems. Despite adopting new platforms and management frameworks, the stress on operations teams persists, and ticket queues are identified as a major contributor to inefficiency. Ticket queues introduce delays, miscommunication, bottlenecks, siloed work, and management overhead, obscuring the value stream and inhibiting learning and feedback loops. While tickets can be useful for specific tasks like logging exceptions or documenting communication, their overuse in complex knowledge work is costly. To address these issues, organizations are encouraged to redesign workflows to minimize handoffs, adopt self-service automation, and eliminate unnecessary ticket queues, thus improving efficiency and reducing operational strain.

Many users often underutilize software by focusing only on its basic features, missing out on the full scope of its capabilities, as illustrated by the author's personal experiences with tools like YouTube and Slack. This phenomenon is common across various software platforms, including PagerDuty, where customers may only use it for a single purpose like on-call management, unaware of its broader digital operations solutions. Recognizing this gap, PagerDuty University aims to enhance customer education by offering training that highlights the full range of PagerDuty's features, from pre-incident event management to post-incident automation, thus maximizing the return on investment for customers. By tailoring training to specific business challenges and providing both instructor-led and on-demand options, PagerDuty University helps users efficiently leverage the platform's advanced capabilities, such as Event Intelligence and Incident Workflows, ultimately driving tool consolidation, cost savings, and improved customer experience.

The text discusses the concept of "toil" in the context of Site Reliability Engineering (SRE) and IT operations, emphasizing its significance as non-value-adding work that is necessary but can hinder progress and creativity within organizations. Originating from Google, toil is described as repetitive, manual, and automatable tasks that grow linearly with service expansion but do not add enduring value. While eliminating toil entirely is unrealistic, reducing it is crucial for maximizing the value of engineering work and maintaining organizational efficiency. High levels of toil can lead to burnout, errors, career stagnation, and operational inefficiencies, thereby impacting both individuals and organizations negatively. To alleviate toil, engineering work is required to automate or enhance systems, but this can create a paradox where toil itself consumes the time needed for such engineering efforts. The text highlights tools like PagerDuty Process Automation and Rundeck that help reduce toil by standardizing procedures and enabling self-service, ultimately aiming to improve the work-lives of operations professionals by freeing up time for more strategic and innovative tasks.

In memory of Phylicia Jones, over 20 employees from PagerDuty, known as Dutonians, gathered in San Francisco to assemble hygiene kits for Compass Family Services, an organization supporting families impacted by economic changes and COVID-19. This initiative resulted in the assembly of over 50 hygiene kits containing essentials like shampoo, soap, and toothbrushes, part of a larger 2022 effort that has provided more than 150 kits. The project, which saw the involvement of in-person volunteers and financial contributions amounting to over $500, exemplifies PagerDuty's commitment to community values such as #Ack&Own and #RunTogether. The company plans to continue these efforts throughout the fiscal year to aid other local communities, marking one of the first major in-person volunteer initiatives since returning to the office post-2021.