When a production deploy fails, our merge queue freezes itself

When a production deployment fails at Mergify, a 66-line GitHub Actions job now freezes the merge queue to prevent further merges onto a broken release. This change was implemented after realizing that failed deployments were not automatically halting subsequent merges, which led to inconsistencies between different deployment scopes like the engine backend and dashboard frontend. The freezing mechanism activates when any of the five deployment scopes fail, blocking further merges until a human intervenes to lift the freeze, thereby ensuring that the issue is addressed before additional changes are deployed. The system is designed to prevent further complications by stopping the merge queue, which forces the team to focus on resolving the failure without piling on more changes. While the solution does not automatically rectify failed deployments, it serves as a loud alert that something is wrong and needs immediate attention. This process is deliberately human-mediated to ensure that the problem is fully understood and resolved before normal operations resume, maintaining the integrity and reliability of the deployment pipeline.

No tracked trend matches for this post yet.