Runs list and logs degraded

Trigger.dev experienced a significant incident on June 30, lasting approximately six and a half hours, during which their dashboard, runs list, and logs were either slow or unavailable due to issues with the ClickHouse database, although run executions remained unaffected. This problem stemmed from a recent automatic upgrade of the ClickHouse cluster, which introduced a new data-type complexity limit that disrupted background merges of JSON columns used for storing task attributes, leading to memory ceiling breaches and query failures. To address the issue, Trigger.dev scaled their database, detached un-mergeable parts, and coordinated with ClickHouse for a service-level fix. They have outlined several strategies to prevent future occurrences, including simplifying JSON types, enhancing alert systems, separating read and write operations, and implementing safeguards against high-complexity data, while acknowledging the need for improved reliability and communication with their users.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.