Semaphore CI/CD Benchmark: Performance and Cost Analysis

A benchmark comparison was conducted to assess the performance and cost efficiency of Semaphore against GitHub Actions, GitLab CI, Buildkite, and CircleCI using the Redmine Ruby on Rails application under identical conditions, focusing on single-job execution speed without parallelism. The evaluation involved measuring execution time and compute cost after cache warm-up across various providers, each using 2 vCPU machines with similar memory configurations. Semaphore emerged as the most efficient, delivering the fastest average execution time of 5:01 minutes and the lowest cost per run at $0.04, in contrast to its competitors, which had longer execution times and higher costs. This performance allows for significant time savings in feedback cycles, which can translate into regained engineering hours and reduced compute waste. The study highlights the importance of considering CI performance not only in terms of direct compute costs but also in terms of the broader impact on engineering productivity and budget efficiency, emphasizing that organizations should use these results as a reference while assessing their own CI pipelines.

No tracked trend matches for this post yet.