Post-Mortem: Incident Report 08/07/2025

A post-mortem report on a Sandbox Archival and Creation Latency Incident that occurred on August 7, 2025, details a series of failures and resolutions over a 22-hour period. The incident was primarily caused by an overwhelming backup policy that led to unsustainable loads, affecting sandbox creation and archival processes. This resulted in 2.3% of sandbox creations failing and 6,389 sandboxes not archiving, impacting four customers. The investigation revealed that the high number of backup operations per runner, combined with a broken image cleanup protocol and storage system bottlenecks, strained the infrastructure. Despite various troubleshooting efforts, including scaling storage systems and rebooting instances, a critical realization was made that the issue stemmed from load management rather than Docker corruption. The incident underscored the need for improved monitoring, rate-limited backup policies, and automated image cleanups, with action items set to address these areas.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.