Service Outage Postmortem: April 28

On April 28th, 2023, Semaphore experienced a service disruption lasting over three hours due to delayed job processing caused by low-performing database queries that led to a CPU usage spike in the production database. The incident began at 22:23 UTC and was resolved by 01:43 UTC after the on-call SRE team implemented a solution involving a manual vacuum operation on the jobs database table. The root cause was identified as the accumulation of dead tuples in a high-read/write DB table, compounded by the auto-vacuuming function not executing properly due to increased traffic. In response, Semaphore plans to enhance database performance monitoring, conduct regular performance testing, and improve incident communication protocols. They acknowledged a breach in their incident response policy due to a failure in updating their public status page, and they committed to revising their internal procedures and training to ensure better communication during future incidents.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.