Update on the March 24, 2016 DNS Outage

DigitalOcean's DNS infrastructure was unavailable for two hours and four minutes due to a DDoS attack. The resolvers were overwhelmed by an unusually large number of queries, causing cache invalidation and further stress on the system. DigitalOcean worked with their DDoS mitigation provider to identify the source and types of traffic, block malicious requests, and increase TTLs for cached DNS records. After engaging with their provider, the service began responding to queries with normal latencies, and most of the filtering was removed by 7:22 pm UTC. To prevent similar incidents in the future, DigitalOcean is reviewing and improving various pieces of infrastructure, including better means of moving DNS traffic around within their network, decoupling provisioning from the DDoS mitigation provider, and enhancing internal communication to test mitigation techniques more effectively.