August 2024 Summaries

On August 15, 2024, a significant spike in CPU usage in the primary database led to prolonged processing times and failure of some database transactions, causing a backlog in queues and a "System failure" status for certain processes. The root cause was identified as an issue with Postgres vacuuming, specifically with the xmin horizon, which prevented dead tuples from being cleaned, thus exacerbating the problem during sequential scans by Graphile Worker. Despite attempts to resolve the issue by terminating transactions and adjusting configurations, the problem persisted until a failover to a read replica was executed, which restored normal operations. The incident highlighted the need for better monitoring of dead tuples and vacuuming processes, and the importance of considering failover as a viable solution to similar issues in the future. Steps are being taken to improve transactional guarantees between Redis and Postgres and to enhance monitoring systems to prevent recurrence.

In August 2023, Trigger.dev released version 2 of its platform, which initially saw strong adoption as it aimed to help developers create long-running background jobs on serverless infrastructure. However, it soon became apparent that this approach led to complex code and numerous issues, prompting the development of version 3, which focuses on simplicity and reliability by removing execution management and timeout concerns. With version 3 now surpassing version 2 in adoption, Trigger.dev announced that version 2 will be discontinued on January 31st, 2025, urging users to migrate to version 3 or consider other solutions. To assist with the transition, Trigger.dev offers a detailed migration guide and support, while also introducing the possibility of self-hosting version 2. The company remains enthusiastic about the potential of version 3, which is positioned as the future of the platform with a robust architecture designed for long-term scalability.