October 2014 Summaries

PagerDuty employs a comprehensive and multi-faceted approach to monitoring its systems, ensuring reliability for the millions of alerts it delivers monthly. Arup Chakrabarti, PagerDuty's engineering manager, detailed the company's monitoring strategies at DevOps Days Chicago, highlighting the use of various tools like New Relic for application performance management, StatsD and DataDog for customizable metrics monitoring, and SumoLogic for log analysis. The company also utilizes external monitoring services such as Wormly and Monitis, alongside its own platform, to consolidate alerts. Emphasizing a focus on cluster-level metrics over single-host monitoring, PagerDuty avoids the brittleness of monitoring individual servers by funneling alerts through a robust monitoring system. They also perform dependency monitoring on third-party SaaS systems with a combination of manual and automated checks, exemplified by their SMS delivery testing framework. Chakrabarti stresses the importance of alerting on meaningful metrics that impact customers, with internal exercises like "Failure Friday" validating the effectiveness of alerts by intentionally disrupting services to test monitoring systems.

Aruba Networks has joined PagerDuty's partner ecosystem, marking the 100th platform integration for PagerDuty, enhancing incident management and operational performance. Aruba's ClearPass access management system now integrates with PagerDuty, allowing customers to quickly identify and respond to network security issues, ensuring reduced customer impact. The integration facilitates end-to-end incident management, offering customers peace of mind by escalating incidents until addressed, as emphasized by Cameron Esdaile, Aruba Networks' senior director of emerging technologies. This partnership also benefits businesses like casinos by improving customer experiences, such as identifying VIPs and managing login issues. With 100 integrations available, PagerDuty's platform offers seamless operations management by connecting to various infrastructure, application, and business tools, and the company plans to continue expanding its integration partnerships.

Blameless postmortems are an effective strategy for addressing failures in complex systems by focusing on understanding how mistakes occur rather than assigning blame to individuals. Engineering managers Arup Chakrabarti and Mike Panchenko emphasize the importance of maintaining a mindset centered on customer impact and problem-solving, rejecting the simplistic approach of blaming individuals for errors. They highlight that failures often stem from systemic issues, and the aim is to learn from these events and improve processes. Both managers stress the need for immediate intervention when blame occurs and for separating ongoing performance issues from systemic failures. They also underline the necessity of executive support to implement successful blameless postmortems, as this approach requires more effort than simply assigning blame. The goal is to foster a culture where employees feel safe to admit mistakes and contribute to resolving them, leading to more resilient systems and processes.

PagerDuty is committed to supporting community initiatives and has actively participated in events such as Movember and Breast Cancer Awareness Month to make a positive impact. Recently, to honor a breast cancer survivor within their team, they incorporated fundraising activities like raffles and poker games into their weekly gatherings, with proceeds going to breast cancer charities. Additionally, they volunteered at a Habitat for Humanity site and are planning a mustache-growing competition for the Movember Foundation next month. Looking ahead to the holiday season, they intend to organize a food drive for the SF-Marin County Food Bank, and they are open to suggestions for other charitable activities.

FutureStack14, held at Fort Mason with scenic views of the Golden Gate Bridge and Alcatraz, provided an engaging platform for attendees to connect with industry peers, including New Relic and PagerDuty customers. Lew Cirne's keynote highlighted the goal of enhancing user experiences by reducing software-related frustrations like lag, errors, and downtime, which can lead to significant annual dissatisfaction. New Relic introduced new Insights capabilities, enabling business users to easily analyze data for actionable insights such as marketing campaign effectiveness and feature usage, while PagerDuty emphasized the importance of using data to improve operational efficiency and reduce software impact on customers. The conference also featured a webinar announcement focused on decreasing mean time to resolution (MTTR) and addressing alert fatigue, underscoring the collaborative efforts between New Relic and PagerDuty to optimize software performance and reliability.

Tim Yocum, Ops Director at Compose, discusses the importance of alert management in handling database services like MongoDB and Elasticsearch, emphasizing the challenges that arise as companies scale. He notes that many companies lack mechanisms for generating and handling alerts in their applications, leading to increased complexity and strain on personnel. Using Compose's experience as a case study, Yocum explains how they manage alerts with tools like PagerDuty, Nagios, and Sensu, integrating these systems to ensure timely and efficient responses to issues. By employing a structured alert management system, Compose can effectively manage and distribute workload among support staff, enhancing their ability to focus on delivering high-quality support without being bogged down by unnecessary disturbances. The blog suggests that businesses can benefit from using Compose and PagerDuty to streamline their alert systems, enabling them to concentrate more on their core operations.

PagerDuty has introduced customizable incident status update notifications to address user frustration when incidents are resolved or acknowledged by others without their knowledge. This new feature allows users to be alerted when an incident is escalated, resolved, or acknowledged by another team member, which is particularly useful in multi-user escalation policies. Users can customize their notification preferences by logging into their PagerDuty account and adjusting settings in the "Status Updates" section of their profile. This enhancement aims to improve communication and efficiency by ensuring users are promptly informed about changes in incident status.

Creating a sustainable data-driven culture in IT operations involves more than just declaring a reliance on data; it requires a strategic approach to selecting and using the right metrics to align with business goals. Managers should understand executive priorities and choose metrics that enhance decision-making without getting bogged down in irrelevant details. Key steps include democratizing data access, empowering team members to engage with data, and continuously testing and acting on insights. A successful data-driven culture encourages open dialogue, where team members feel comfortable questioning decisions and proposing actions based on data. Avoiding common pitfalls such as focusing too heavily on past metrics, managing solely to numbers, and succumbing to analysis paralysis is crucial, as is maintaining a feedback loop where insights lead to informed actions and ongoing discussions.