End-to-end reliability testing with PagerDuty & Datadog

At PagerDuty, we use Datadog to measure the effectiveness of third-party services that deliver SMS alerts to our customers. We can't control these providers, but by combining Datadog and PagerDuty, we've created a practice that proactively discovers outages in one of our provider's systems, quickly finds a replacement, and minimizes customer impact. This process involves setting up multiple phones with different mobile carrier networks, using an internally-built mobile app to send SMS alerts, and analyzing the time taken for each SMS to reach the designated phone and how long it takes to reply back. When a provider exceeds our acceptable thresholds, we consider it downgraded and take action to switch priority levels of each provider to ensure the most functional one is used first.