April 2017 Summaries

DigitalOcean recently experienced a significant outage in their SFO2 region, affecting compute and networking services. The incident was caused by the failure of multiple redundant power distribution units (PDU) within the datacenter, which led to intermittent failures of the control panel and API. The recovery effort was prolonged due to heavy load on the event processing system, which was unable to support the load of the SFO2 datacenter recovery in addition to normal operational load from other datacenters. To prevent similar incidents in the future, DigitalOcean is implementing stronger safeguards and validation of their power management system, reviewing datacenter recovery procedures, and adding additional capacity to their event processing system. The company also plans to issue SLA credits to affected users.

The outage occurred on April 5th, 2017, at 10:24 AM EDT, and lasted four hours and fifty-six minutes. The primary database was deleted, but data was restored within the duration of the outage. The root cause was an engineer-driven configuration error caused by misconfigured production credentials. DigitalOcean is taking steps to improve network connectivity, hardware, and processes to prevent similar incidents in the future.

Monitoring is now available for free with all Droplets, providing developers with tools to resolve issues quickly by alerting them when an issue occurs and giving them information to understand it. The service is natively integrated with the DigitalOcean platform, can be enabled at no extra cost, and introduces new alerting capabilities using metrics collected by the agent. It exposes system metrics and provides an overview of Droplets' health, allowing teams to receive notifications based on resource utilization and operational health. The monitoring service offers features such as creating alert policies, leveraging tags to group Droplets, and installing the agent on existing Droplets. The team is working on future iterations that will include features such as visualizing metrics and integrating Monitoring into workflows.