April 2017 Summaries

Semaphore introduces a "fast failing" feature designed to enhance developer productivity by providing instant feedback when a job fails, thereby minimizing context switching and maintaining a state of flow. This approach stops all running jobs of a build as soon as a failure is detected, allowing developers to address issues more swiftly without waiting for all tests to complete. For instance, if a build takes 10 minutes, the fast failing feature reduces the total time to 12 minutes compared to 21 minutes without it, thanks to more immediate feedback and quicker issue resolution. Developers can enable fast failing through branch settings, choosing its application across all branches or excluding the default one, thus optimizing their workflow efficiency.

Flaky tests, which unpredictably pass or fail, are often handled by teams with short-term solutions like rerunning them until they pass, but this approach is criticized for encouraging long-term test suite degradation and masking underlying issues in the code or tests themselves. While some managers prefer this quick fix due to immediate corporate pressures, the practice is seen as dangerous because it assumes the problem lies within the test rather than the production code, potentially affecting customers. A more sustainable solution involves fixing or rewriting flaky tests, although this can be time-consuming and aggressive. However, there are instances where rerunning tests may be justified, such as when time-to-market is critical and technical debt is managed with a clear plan to resolve it. Ultimately, the choice to rerun flaky tests impacts customer perception and the integrity of the testing process, leading some to argue against it due to its potential harm.

Semaphore previously enforced a 60-minute execution limit on build commands, which sufficed for most projects but proved inadequate for tasks such as compiling large binaries, provisioning extensive infrastructures, or running complex tests. To address the need for flexibility, Semaphore introduced a new configuration option in a project's admin section, allowing developers to set a custom timeout that better aligns with their specific project requirements. This change aims to balance the need for extended build times with the desire to prevent indefinite execution due to issues like accidental debug statements or stalled network calls.