What happened during our AWS outage

From May 14 to May 18, 2026, Helicone experienced an observability outage due to AWS flagging one of their Amazon Bedrock API keys as possibly compromised, which resulted in the locking of their entire AWS account. This action prevented ECS tasks from starting, affecting log ingestion, the dashboard backend, and dashboard authentication, while the Helicone proxy remained functional, ensuring that user traffic was not disrupted. Although no data was lost, and requests during the outage were queued for later processing, the incident resulted in a four-day backlog due to the time taken by AWS to respond and lift the block. Despite confirming that the flagged key was not compromised, Helicone rotated all credentials as a precaution, and services were restored to normal by May 18, with the backlog expected to clear gradually. The company apologized for the extended duration of the outage and encouraged users to reach out if data appeared incomplete after the backlog is processed.