Incident report for September 18, 2026 - When Deletion Cascades Exhausted Our Database Connections
Blog post from Inngest
On September 18, 2026, a Vercel Marketplace account deletion triggered two platform-wide service disruptions after a hard delete cascaded across dozens of PostgreSQL tables, holding locks for an extended period while repeated deletion requests for the same account queued behind it. The blocked queries retained PgBouncer client connections, exhausting the shared connection pools and preventing unrelated services from accessing the database between approximately 16:18–16:42 UTC and again from 17:50–18:12 UTC, affecting run scheduling, event acknowledgements, APIs, queue processing, and several services that entered crash loops. Restarting PgBouncer did not resolve the issue while the blocking transactions remained active, and recovery in the second incident lagged after database locks cleared because queued connections still needed to drain. Accepted events were stored durably and delayed work later caught up, though users who did not receive an acknowledgement are advised to retry their event. The company has disabled marketplace deletion operations through a kill switch, plans to replace hard deletion with soft deletion where possible, move necessary hard deletes into serialized batched jobs, improve incident rollout procedures, add PgBouncer capacity and isolation, and expand monitoring for database locks and connection waits.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.