Postmortem: Clerk System Outage (February 19, 2026)
Blog post from Clerk
On February 19, 2026, Clerk experienced a service outage due to a "query plan flip" caused by Postgres' automatic analyze function, leading to a significant degradation in database performance as it miscalculated the percentage of NULL values in a column. The inefficient query plan resulted in over 95% of traffic returning 429 errors without being handled, while the few requests reaching the database were processed extremely slowly. The issue was resolved approximately 90 minutes later by manually re-running the ANALYZE command, restoring the query to its prior plan and normalizing database performance. During the incident, Clerk identified and attempted to mitigate a coincidental traffic spike from a customer's aggressive retry mechanism, and implemented a new failover mechanism to handle session token generation, which temporarily eased the load. In response, Clerk plans to improve alerting for query plan flips, harden session token failovers, enhance query plan stability, and formalize incident communication processes to prevent similar issues in the future and improve customer communication during incidents.