Company
Date Published
Author
Fantix King
Word count
1508
Language
English
Hacker News points
None

Summary

EdgeDB significantly optimized its continuous integration (CI) workflow by utilizing GitHub Actions to parallelize a comprehensive suite of tests, achieving a runtime reduction from over two hours to just 10 minutes. The test suite, which consists of approximately 5000 tests and 60 database setups, was split into shards using a custom algorithm that allocates tests into evenly distributed buckets to maximize efficiency. The process involves creating priority queues for both standalone and setup-linked tests, and strategically allocating them to ensure minimal setup initialization and efficient resource use. By employing a distributed computing approach, each runner independently executes the sharding algorithm, thereby avoiding the complexities of having a single control arbiter. EdgeDB's solution includes a final integrity check to ensure all tests are executed and uses a CSV file for tracking test runtimes, which is updated after each CI run. This innovative use of GitHub Actions demonstrates its versatility for managing complex workloads efficiently, resulting in a CI process that is approximately 95% faster.