Flaky Tests: Are You Sure You Want to Retry Them?

Flaky tests, which unpredictably pass or fail, are often handled by teams with short-term solutions like rerunning them until they pass, but this approach is criticized for encouraging long-term test suite degradation and masking underlying issues in the code or tests themselves. While some managers prefer this quick fix due to immediate corporate pressures, the practice is seen as dangerous because it assumes the problem lies within the test rather than the production code, potentially affecting customers. A more sustainable solution involves fixing or rewriting flaky tests, although this can be time-consuming and aggressive. However, there are instances where rerunning tests may be justified, such as when time-to-market is critical and technical debt is managed with a clear plan to resolve it. Ultimately, the choice to rerun flaky tests impacts customer perception and the integrity of the testing process, leading some to argue against it due to its potential harm.