July 2026 Summaries

On July 21, 2026, Blacksmith’s control plane suffered major degradation from about 10:20 AM to 3:55 PM ET after a burst of erroneous, largely duplicate GitHub webhook events overwhelmed an already heavily loaded Redis instance used to process and queue customer jobs. Redis CPU saturation slowed requests, exhausted HTTP workers, caused cache, monitor, and sticky-disk operations to fail or time out, slowed job execution, and created a substantial backlog of unstarted jobs; most regions returned to normal queueing by 5:25 PM ET, eu-central recovered around 7:00 PM ET, and failed dispatches were reissued by 7:19 PM ET. Recovery was prolonged when an attempted HTTP worker scale-up exhausted database connections and shared-pool health checks caused the load balancer to terminate healthy instances, obscuring the Redis root cause and reducing available capacity. Blacksmith resolved the incident by distributing Redis traffic across multiple instances and moving load-balancer checks to an unaffected status endpoint, and it plans to strengthen independent health checks, monitoring, alerting, capacity targets, and broader control-plane architecture. The company also acknowledged inadequate customer communication and committed to a formal incident process with a communications owner and status updates at least every 30 minutes.