Outage Post-Mortem – Dec 11, 2013

PagerDuty experienced an outage on December 11th due to a failure in their DNSSEC infrastructure, specifically related to the expiration of a Resource Record Signature (RRSIG) that was not renewed due to a fault in their DNS provider's automation system. This affected a subset of customers who used DNS providers enforcing DNSSEC, preventing access to pagerduty.com addresses, APIs, and apps. The outage highlighted weaknesses in PagerDuty's monitoring setup, which had a networking bias rather than a security focus, causing delayed detection of the issue. In response, PagerDuty plans to enhance their infrastructure by updating monitoring protocols, setting up continuous RRSIG expiration monitoring, reducing DNS record Time-to-Live (TTL) to minimize propagation delays, and establishing a secondary DNS provider to increase redundancy. Despite the outage impacting less than 2% of their total request volume, it underscored the need for improved security and monitoring practices, especially as more DNS providers adopt DNSSEC enforcement.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.