December 2013 Summaries

PagerDuty plays a crucial role in enhancing incident management and on-call scheduling for a variety of companies, as reflected in experiences shared by several customers. Panasonic uses PagerDuty to significantly reduce incident response times by implementing it for both IT and customer support on-call scheduling, allowing teams to be alerted only when issues arise. GREE has integrated PagerDuty into its DevOps model, reducing resolution times by routing incidents to the developers responsible and eliminating the human factor in escalation. Zeebox leverages PagerDuty for non-technical teams, ensuring smooth scheduling and incident management for content producers and editors, who coordinate live commercials with their app. Wooga, a data-driven organization, uses PagerDuty to reliably receive alerts and manage incidents, helping them maintain their social games' online availability and track response and resolution metrics.

PagerDuty experienced an outage on December 11th due to a failure in their DNSSEC infrastructure, specifically related to the expiration of a Resource Record Signature (RRSIG) that was not renewed due to a fault in their DNS provider's automation system. This affected a subset of customers who used DNS providers enforcing DNSSEC, preventing access to pagerduty.com addresses, APIs, and apps. The outage highlighted weaknesses in PagerDuty's monitoring setup, which had a networking bias rather than a security focus, causing delayed detection of the issue. In response, PagerDuty plans to enhance their infrastructure by updating monitoring protocols, setting up continuous RRSIG expiration monitoring, reducing DNS record Time-to-Live (TTL) to minimize propagation delays, and establishing a secondary DNS provider to increase redundancy. Despite the outage impacting less than 2% of their total request volume, it underscored the need for improved security and monitoring practices, especially as more DNS providers adopt DNSSEC enforcement.

The integration partnership between Dell Foglight and PagerDuty aims to enhance application monitoring and incident management for organizations worldwide. Foglight provides real-user experience monitoring to identify application issues and customer behavior patterns, such as why users abandon transactions, thereby improving user satisfaction and supporting organizational IT needs. It helps diagnose why customers might not complete purchases and ensures high website performance by alerting users to database CPU issues through various notification methods like SMS, phone calls, push notifications, or email. This integration streamlines the on-call process with set-and-forget schedules and auto-escalation rules, making incident management more efficient and ensuring immediate issue resolution. Users can install the PagerDuty cartridge to start using these features, thereby maintaining smooth and reliable IT operations.

Creating a thoughtful gift for an on-call colleague or family member during the holidays can be both practical and appreciated, particularly with an "on-call survival bag" designed to address the unique challenges of being on call. Essential items include extra laptop and phone chargers to ensure devices remain operational, a MiFi mobile hotspot for reliable internet access, breath mints for maintaining freshness in close quarters, headache remedies like NSAIDs or tea to alleviate stress-induced discomfort, and a modern smartphone for efficient communication using apps like PagerDuty. The bag itself can be personalized to suit the recipient's style, and additional items like a laptop, external monitor, non-perishable snacks, and energy drinks can further enhance the experience, providing both functionality and a touch of holiday cheer.

A study conducted on PagerDuty customers using either Android or iPhone mobile apps found that Android users generally resolve IT incidents faster than their iPhone counterparts, with a Mean Time to Repair (MTTR) of 119 minutes compared to 225 minutes for iPhone users, marking a 52% speed advantage. However, when focusing on incidents resolved within 24 hours, the difference narrows to 20%, with Android users resolving incidents in 42 minutes compared to 48 minutes for iPhone users. For incidents resolved in under an hour, the disparity is negligible, with Android users being faster by just one second. This analysis raises questions about whether device preference influences IT response efficiency or if other factors may be at play.

Emails have evolved from casual communication to being a crucial tool for official correspondence, record-keeping, and identity verification, despite their reputation for being less immediate than texts or tweets. They serve as a permanent, searchable repository for vital information such as bank statements and legal notices, and enable detailed communication through their capacity to include comprehensive descriptions and links. While the volume of emails, including spam, can be overwhelming, tools like PagerDuty enhance their functionality by consolidating and routing alerts effectively to ensure timely and accountable responses. Furthermore, emails can be integrated with task management systems, allowing them to function as actionable to-do lists that are easily accessible and organized. However, while ideal for office work, their effectiveness in urgent situations outside work hours can be limited, prompting discussions on alternative alert methods like SMS and phone call alerts.

PagerDuty's Web team is dedicated to delivering a seamless user experience by ensuring their customer-facing features are reliable, fast, and intuitive. Under the leadership of Web Engineering Manager Amy Chantasirivisal, the team recently launched a new mobile app, demonstrating their commitment to innovation and customer satisfaction. Team members such as Greg Hoin, a User Experience Engineer and Product Designer, focus on understanding user needs and building responsive products, while Software Engineer Will Bronitsky takes pride in his career transition and enjoys engaging in various leisure activities. Daniel Ribeiro, known as a "hacker" at PagerDuty, contributes to the development of all user-interactive platforms, and Clay Smith, the resident bartender, is passionate about crafting spirit-driven cocktails. Steve Rice, who specializes in refining the implementation of features, emphasizes developing straightforward solutions to complex challenges, while the diverse interests and unique talents of the team members, from karaoke enthusiasm to typeface recognition, enrich their professional environment.

The series highlights the personal experiences of PagerDuty's on-call team members, focusing on the effectiveness of different alerting methods during system issues. Among the contact methods discussed, phone calls are emphasized for their ability to convey urgency and immediacy, cutting through communication clutter and ensuring quicker responses, especially during high-severity incidents. Despite the decline in phone calls due to the rise of SMS and other digital communications, their rarity and directness make them a valuable tool for alerting, as they are less likely to be ignored than emails or text messages. The series also notes that while phone calls may interrupt social settings, they are crucial for waking up on-duty staff, as highlighted by Mozilla service engineer Benson Wong's reliance on phone call alerts to wake him from sleep, thus reducing response time and the need for escalation.

In 2014, Gartner highlighted "software-defined anything" (SDx) as a key strategic technology, emphasizing the importance of automation in cloud computing, DevOps, and infrastructure provisioning. Inkling, an interactive ebook platform, exemplified this trend by transitioning to a DevOps model to enhance their software development process and meet customer SLAs. Their shift involved adopting monitoring tools to automate issue detection and using PagerDuty to streamline incident management by centralizing alerts and automating on-call schedules. This transformation allowed Inkling to deliver a reliable service, facilitating a mobile-first, digital-first publishing model. Inkling's DevOps strategy has supported their growth and innovation, positioning their platform, Inkling Habitat, as a leading cloud-based solution for building interactive content used by top global publishers.