July 2026 Summaries

Trigger.dev experienced a significant incident on June 30, lasting approximately six and a half hours, during which their dashboard, runs list, and logs were either slow or unavailable due to issues with the ClickHouse database, although run executions remained unaffected. This problem stemmed from a recent automatic upgrade of the ClickHouse cluster, which introduced a new data-type complexity limit that disrupted background merges of JSON columns used for storing task attributes, leading to memory ceiling breaches and query failures. To address the issue, Trigger.dev scaled their database, detached un-mergeable parts, and coordinated with ClickHouse for a service-level fix. They have outlined several strategies to prevent future occurrences, including simplifying JSON types, enhancing alert systems, separating read and write operations, and implementing safeguards against high-complexity data, while acknowledging the need for improved reliability and communication with their users.