Post-Mortem: Feb 16 2022

On February 16th, 2022, Authzed noticed consistently higher API latency with occasional spikes into the hundreds of milliseconds due to new metrics features in their billing section. The root cause was determined to be always present and made observable once fully exercised with the introduction of the new functionality. After a full deployment rollback, the service returned to its previous stability, but the root cause was still identified as being related to caching issues in SpiceDB, specifically the thundering herd problem caused by traffic patterns. The issue was resolved by optimizing SpiceDB to be more conscious of its impact on the Go scheduler and increasing the number of cores available to SpiceDB. Following the resolution, overall API latency improved 27%.