August 2011 Summaries

PagerDuty has shifted from using Twitter to a status page for communicating about major outages, following a significant AWS cross-AZ failure in the US-EAST region. To enhance communication during large-scale issues, they are developing an opt-in notification system that is separate and external. Meanwhile, they advise users to subscribe to their PagerDuty Status page for updates. Although they plan to rarely use Twitter for updates, users can still receive direct notifications from the PagerDuty Ops team by enabling mobile notifications for the @pagerdutyops Twitter account. Detailed instructions are provided for setting up these notifications, emphasizing the importance of staying informed about potential disruptions in PagerDuty's notification services.

PagerDuty experienced a 30-minute outage due to a simultaneous failure across three independent Amazon Web Services (AWS) Availability Zones in the US-East-1 region, which interrupted their ability to process and dispatch notifications. This event exposed vulnerabilities in their infrastructure and highlighted the need for greater redundancy and load management. In response, PagerDuty has implemented immediate measures, such as deploying a replica of their stack with an additional hosting provider and increasing front-end capacity, to prevent similar outages. They plan to shift away from AWS entirely and host across multiple providers to minimize correlated failure risks, and they are enhancing their load testing to handle high-traffic scenarios effectively. Additionally, PagerDuty aims to improve its customer communication during outages by introducing a Twitter account for downtime notifications and exploring phone alert systems. These efforts are part of a broader strategy to ensure reliability and regain customer trust.

From August 8 to 10, PagerDuty is attending HostingCon 2011 in San Diego, a premier event for companies in the website hosting industry, where they aim to connect with current and potential customers while launching a new partnership program for hosting providers. The company plans to engage attendees with a booth (#450) featuring a unique giveaway contest, where participants can win prizes such as iPad 2s and Amazon gift certificates by quickly responding to SMS alerts sent to their mobile devices. Additionally, attendees can visit the booth for PagerDuty-branded t-shirts and other swag, making the event both a networking and promotional opportunity.