Company
Date Published
Author
Alex Ntousias
Word count
287
Language
English
Hacker News points
None

Summary

On August 28, a distributed credential stuffing attack on a specific tenant's authentication endpoints caused two brief periods of elevated latency and errors in both the Frontend and Backend APIs, although services remained partially available. The incident, which lasted approximately 22 minutes and 12 minutes during two separate windows, was mitigated without disruption to session token issuance. The root cause was identified as a combination of factors, including automated traffic generating high volumes of sign-in and sign-up attempts, write-intensive activity increasing contention on hot authentication-related tables, and a lagging CDC consumer under burst conditions. Errors affected 2.52% of Backend API requests and 0.14% of Frontend API requests, but there was no data loss or corruption. Remedial actions include disabling the lagging change-stream processor, strengthening adaptive protections at the edge and auth layer, and improving schema and query-path to reduce contention. Additionally, efforts will be made to enhance per-customer isolation to limit the impact of such issues in the future.