January 2014 Summaries

PagerDuty experienced an outage on January 23, 2014, due to a slow database query linked to new mobile app functionality, which caused high load on a database server, resulting in delayed notifications and some event loss for users over an 18-minute period. The team promptly addressed the issue by terminating the problematic queries and rolling back the mobile app version, while also removing backend functionality to mitigate the problem for iOS users due to slower app publishing processes. In response, PagerDuty has refactored the code to improve query performance significantly and implemented a slow query killer to proactively manage similar issues. Moving forward, they plan to enhance database query audits, perform more rigorous performance testing, optimize server configurations, and increase monitoring to prevent future incidents, all while committing to transparency and reliability for their customers.

A recent survey of on-call engineers revealed that while many enjoy spending time with friends and family during non-business hours, a significant portion feels confined to their homes due to their responsibilities. The article suggests various activities to make on-call time more productive and enjoyable, such as engaging in physical activities like going to the gym or bowling to counteract the adverse effects of prolonged sitting, playing video games to enhance scientific thinking, or catching up on household chores. Additionally, it encourages on-call engineers to pursue entrepreneurial projects or spend time with loved ones, with the reassurance that critical issues will be communicated by PagerDuty if necessary. These suggestions aim to help on-call engineers balance their responsibilities with personal time, encouraging them to share their own experiences in the comments.

PagerDuty offers a vibrant and engaging work culture characterized by unique traditions such as Monthly Hack Days, Whiskey Wednesdays, and video game nights, alongside quirky elements like a snack area named SnackDuty and conference rooms inspired by Futurama characters. Nisha, a new member of the team, expresses her enthusiasm for joining this innovative environment, emphasizing the authentic and geek-friendly culture that aligns with her own background in computer science and passion for technology. As a marketer and self-proclaimed geek, she is eager to contribute to PagerDuty's mission of enhancing the lives of IT professionals, developers, and operations teams, while also increasing her own "geek quotient" by working alongside her knowledgeable colleagues.

Monitoring tools often limit alerts to email, which can be ineffective for on-call personnel. PagerDuty experts suggest using a combination of alert types, including phone, SMS, email, and push notifications, to ensure timely responses. They share methods like the "Boom-Boom-Boom In My Pants" for immediate alerts and the "Minuteman" for staggered notifications, which escalate the urgency. Other strategies include the "Otherwise Silent" and the "Let My Wife (Or Husband) Sleep" methods, focusing on minimizing disruptions while maintaining alert effectiveness. The "Trigger Happy" approach combines multiple alert types with minimal delay to ensure rapid incident acknowledgment. Users are encouraged to customize their notification settings and share their strategies in the comments.

Forgetting about being on-call is a common experience, as evidenced by a survey revealing that over 80% of participants have done so, often when away from the office and engaged in personal activities. The text offers various real-life scenarios and advice on how to handle unexpected alerts while on-call, such as being reminded by friends while out for a drink, buying movie tickets, gaming at home, or attending a concert. It suggests practical solutions like preparing an on-call emergency backpack, contacting a secondary person in the escalation policy, and quickly escalating incidents to ensure prompt handling. The emphasis is on remaining flexible and responsible, while also acknowledging the occasional need to negotiate shift coverage with teammates.

PagerDuty is a tool designed to enhance incident alerting and management by offering customizable notification methods and escalation policies, ensuring that alerts are effectively communicated to the appropriate personnel. Users can tailor their contact methods, such as phone, SMS, email, and push notifications, and establish notification rules to receive alerts in a preferred manner. Escalation policies serve as a backup to re-route alerts if the primary on-call engineer fails to respond, and it is recommended to set up multiple contact methods and notification rules to avoid missed alerts. PagerDuty also implements features to reduce alert fatigue, including deduplication, bundling, and appending of alerts, ensuring that users receive fewer, more relevant notifications. Furthermore, the system automatically resolves open incidents after a set period and re-triggers alerts if an acknowledged incident is forgotten. These capabilities allow PagerDuty to minimize downtime and streamline the resolution process by directing alerts to the right on-call personnel efficiently.

PagerDuty emphasizes its commitment to transparency regarding service outages and advises following their dedicated Twitter account for updates. On January 16th, a minor incident resulted in the delay of six alerts due to a rare race condition caused by efforts to improve service scalability, affecting operations in Cassandra and Zookeeper. Although no alerts were lost, the delays affected three email, two SMS, and one push notification. The issue was swiftly resolved, with regression testing conducted to prevent future occurrences. PagerDuty apologizes for the inconvenience and encourages users to reach out with any questions.

At PagerDuty's Hack Day, a team of engineers and a designer developed "PajamaDuty," an innovative project integrating PagerDuty alerts with wearable technology. This involved embedding a Bluetooth-capable microcontroller into a T-shirt to vibrate and light up when an alert is received, eliminating the need for a phone nearby during sleep. The wearer can acknowledge incidents by tapping their shoulder three times, preventing accidental acknowledgments. The project utilized electronics like a 9V battery pack, LED, electric motor, and conductive fabric, all soldered and stitched into the T-shirt. Alongside, a prototype for matching pajama bottoms was created to merge technology with fashion. Other notable projects from Hack Day included automated GoToMeeting webhooks and an On-Call Dashboard.

PagerDuty is committed to maintaining high service reliability and 100% uptime, with systems in place to ensure any detected events are promptly routed to the appropriate team members via alerts. However, understanding the inevitability of unexpected outages, PagerDuty emphasizes the importance of a contingency plan and transparent communication with users during such events. To keep users informed, the company offers an opt-in Twitter notification system, specifically through the @PagerDutyOps account, to alert users via SMS of any PagerDuty-wide outages. Users can set up mobile notifications to receive updates directly to their phones. Additionally, PagerDuty provides customer support through email, phone, and live chat, and maintains a support Twitter account, @PagerDutyHelp, for addressing minor issues and inquiries. The main Twitter account, @PagerDuty, is used for announcing major product updates and sharing relevant content.

PagerDuty is experiencing an exciting year in 2014, marked by its recent move to a new, upscale office space designed to accommodate business growth and enhance customer service. The new headquarters features 14 conference and telephone rooms named after a popular television show, a fully stocked kitchen with dining areas, three lounge areas for relaxation, and a server room tailored for DevOps needs. The company is enthusiastic about the move and plans to host its first event in the new space on January 16th with HackerX, inviting others to consider the venue for future meetups or co-hosted events.

Push notifications offer a streamlined way to receive alerts about system issues directly on your phone, providing a more actionable and immediate response compared to traditional SMS or phone call alerts. With the PagerDuty mobile app, users can quickly assess the urgency of an incident via push notifications and take necessary actions such as acknowledging, resolving, or escalating incidents without needing a computer. This immediacy allows for faster incident management, particularly useful during inconvenient times like the middle of the night, where a subtle notification can alert the user without disturbing others. The app's design is user-friendly, ensuring accessibility and ease of navigation for rapid response, highlighting the evolving preferences in alerting methods and their practical applications.

In an effort to enhance functionality and user experience, recent updates to the mobile app have introduced several improvements, including the addition of the pull-to-refresh feature and enhanced touch interaction for smoother navigation. Addressing a memory leak issue in older Android devices, the updates ensure the app remains responsive over extended use. Minor user interface changes have been made to simplify feature discovery and improve clarity, such as visual indicators for incident options and renaming certain menu items for better understanding. Users are encouraged to provide feedback on desired future features, while the app remains available for download on Google Play and Apple's App Store.