225 Self-Hosted GitHub Actions Runners: Why We Picked Docker Over VMs

Faced with a rapidly increasing GitHub Actions bill of $400 a day, the team decided to bring their continuous integration (CI) infrastructure in-house using three bare-metal hosts, eventually replacing virtual machines (VMs) with Docker containers. The transition aimed to reduce costs and align with the CI Insights they provide to clients. Initial testing showed significant performance disparities between bare-metal hosts and VMs, prompting extensive optimization efforts with mixed results. Ultimately, the team opted for Docker due to its quicker image iteration and boot times, despite its reduced isolation compared to VMs, as their current needs involved only private repositories without workflow secrets. They addressed challenges like rate limits through a proxy stack and emphasized the importance of image-refresh time over job duration. The static fleet setup enabled predictable costs, contrasting with dynamic autoscaling options, and was managed using a custom declarative reconciler. The transition highlighted areas for improvement, including the need for better runner lifecycle management from GitHub.