February 2019 Summaries

PagerDuty is eagerly anticipating the RSA Conference, where it plans to share new insights aligned with the theme of "Better," focusing on enhancing security through improved processes and collaboration. Recognizing challenges such as alert overload and inefficient response processes faced by security analysts, PagerDuty aims to support its 1000+ security customers and the broader DevSecOps movement by open-sourcing security training and incident response, and by expanding its integrations with over 300 partners across various security domains, including SIEM, SOAR, threat intelligence, and cloud security. Highlighted partners include Aqua Security, CloudGuard Dome9, Palo Alto Networks, Signal Sciences, Templarbit, and Threat Stack, among others, each offering unique solutions for cloud and application security. PagerDuty will also engage with attendees at the conference, offering presentations and demonstrations with partners like Aqua Security, DFLabs, Sumo Logic, Twistlock, and Swimlane, to showcase collaborative efforts in addressing security challenges.

Ana Medina, a Chaos Engineer at Gremlin, participated in a PagerDuty Community AMA, discussing her work in chaos engineering, which involves conducting proactive experiments to prevent outages. She addressed common fears surrounding chaos engineering, emphasizing the importance of monitoring, establishing abort conditions, and starting tests in non-production environments to mitigate risks. Ana also highlighted the role of effective management and self-care in preventing burnout, noting the need for open communication with managers and the value of practicing response scenarios during business hours. Additionally, she expressed enthusiasm for the growing adoption of chaos engineering practices and shared her interest in learning about world history and observability best practices. She further noted that chaos engineers often use tools like PagerDuty to simulate alerts and train for better response to system issues.

PagerDuty, founded in 2009 by Alex Solomon, Baskar Puvanathasan, and Andrew Miklas, celebrated its 10-year anniversary in 2019, having evolved from a bootstrapped startup into a global company with over 500 employees and more than $100 million in annual recurring revenue. Initially inspired by the need for on-call management tools at large tech companies like Amazon, the founders recognized a market gap and quickly launched a minimum viable product, which gained traction through word of mouth in the DevOps community. Early challenges included refining their product and sales model, but joining Y Combinator in 2010 helped them secure seed funding, leading to significant growth and investment from notable firms like Andreessen Horowitz. As PagerDuty expanded, it developed a clear incident management vision and a strong company culture, eventually appointing Jennifer Tejada as CEO in 2016, who further accelerated growth and diversity initiatives. By 2018, PagerDuty had transformed into a comprehensive digital operations platform with new product offerings like Modern Incident Response and Event Intelligence, marking a decade of innovation and setting the stage for future advancements.

PagerDuty has launched its Analytics product, now generally available, which combines machine and human response data to offer operational insights aimed at enhancing process maturity and business outcomes. The product introduces prescriptive operational performance scorecards, developed after studying high-performing organizations, to foster a DevOps culture by encouraging best practices and improving team collaboration. These scorecards are designed to be used in regular operational reviews, facilitating informed decision-making and uncovering unnecessary operational loads. The On-Call Hand-Off Report Scorecard and Service Ops Scorecards aid in evaluating team effectiveness and technical service health, respectively, helping teams respond more efficiently to incidents by providing metrics such as interruptions and downtime. This offering, together with PagerDuty Visibility, provides a comprehensive view of machine data and incident response, enabling effective communication with stakeholders and continuous operational improvement.

Establishing a culture of continuous learning and improvement within an organization involves more than just policy changes; it requires collaboration, persistence, and experimentation, as discussed in PagerDuty's approach to implementing blameless postmortems. The process begins with securing leadership buy-in, which is crucial for fostering an environment where team members can openly discuss failures without fear of blame, thus accelerating problem resolution and system resilience. The article emphasizes the importance of starting small, such as experimenting with one team or smaller incidents, to gradually introduce new practices and build trust among team members. By encouraging transparency and accountability through shared learning experiences, organizations can boost morale, increase employee retention, and enhance productivity. Ultimately, this cultural shift leads to happier, healthier teams and improved software outcomes, as it aligns with Agile and DevOps methodologies that PagerDuty values.

After resolving a major incident, conducting a postmortem is crucial for fostering a culture of continuous improvement and preventing future occurrences. A postmortem involves a detailed analysis of the factors leading to the incident, the response actions taken, and plans to avert similar events in the future. It includes a meeting to discuss findings and share learnings with the organization and stakeholders. Emphasizing a blameless approach, this process encourages open communication without fear of punishment, treating human error as a symptom of systemic issues rather than individual faults. The guide offers insights into overcoming cognitive biases that may cloud postmortem analysis and provides strategies for implementing effective postmortems. By prioritizing postmortem completion soon after an incident, organizations can ensure valuable lessons are learned and integrated into their systems, thus enhancing overall resilience and performance.