July 2012 Summaries

A new integration has been announced between Datadog, a SaaS-based monitoring service, and PagerDuty, which enhances incident management by allowing users to receive notifications via phone, SMS, and email for important events monitored by Datadog. This seamless integration enables the creation and resolution of incidents directly from the Datadog stream, with PagerDuty handling the necessary notifications. It also continuously imports incidents, escalations, and resolutions into Datadog, facilitating discussion and correlation of metrics during or after an incident. For example, incidents triggered by high load peaks on a Cassandra cluster can be visualized alongside actual load graphs. The integration is easily enabled through PagerDuty Connect, with further details available in the Datadog/PagerDuty integration guide.

At PagerDuty, the weekend saw significant disruptions due to a derecho storm affecting 7% of AWS and the addition of a leap second to UTC, both of which caused widespread server issues, though the AWS outage appeared more severe at first glance. While initial graphs showed a 20x increase in traffic related to the leap second, further investigation revealed that the AWS spike hit faster and was 30 times higher than the average traffic during the peak, compared to the leap second's 18 times increase. The AWS outage averaged 7 times higher over two hours, while the leap second spike averaged 9 times higher, highlighting the complexity of interpreting internet incidents and system load. Despite these challenges, PagerDuty's approach to monitoring at the account level through de-duplication and escalation helped manage the situation, with a focus on understanding different alert types providing further insight into the response dynamics.

PagerDuty has announced a new integration with Verelo, a Toronto-based SaaS website monitoring system, which will enhance monitoring capabilities by allowing websites and APIs to be checked as frequently as every 5 seconds. This integration enables users to link their Verelo accounts with PagerDuty via an API, facilitating real-time notifications through phone calls, SMS, and email when websites or APIs experience downtime or issues. The integration process is straightforward, involving logging into both Verelo and PagerDuty accounts, creating a service, and configuring Verelo checks to trigger alerts in PagerDuty. The announcement emphasizes the ease of setting up the integration and offers support for users needing assistance.

On the evening of June 29th, Amazon Web Services (AWS) experienced a significant power outage at its North Virginia location, affecting numerous customers, including those using PagerDuty, although PagerDuty itself remained unaffected. The outage, which was the second in June, led to a notable increase in alert traffic across the PagerDuty platform, with AWS reporting that approximately 7% of EC2 instances in the US-EAST-1 Region were impacted. To mitigate similar risks, PagerDuty had previously migrated its data centers from AWS US-East to US-West to reduce dependency on the same infrastructure used by its customers. PagerDuty emphasizes its commitment to service availability and encourages customers to reach out with any concerns.