April 2023 Summaries

To prepare the ABQ and Captain repositories for open-source release without exposing private issues and pull requests, the team rewrote Git history to replace outdated pull-request references, clean up verbose squash-merge messages, and preserve links to each original commit. Because commit-signature verification was enabled, contributors collaboratively ran a Ruby script that assigned each person responsibility for rewriting commits associated with their email addresses and signing their replacements. The script processed commits from oldest to newest, retained each original tree and mapped parent relationships to preserve identical file contents and history structure, reformatted messages to include original PR and commit identifiers, preserved co-author credit, and maintained an old-to-new hash mapping so it could resume or coordinate with other participants. Contributors pushed rewritten commits to a new branch while the script fetched updates and waited for commits authored by others, allowing the team to generate a fully signed replacement history in minutes. An alternative considered was creating placeholder issues for historical private pull requests, but rewriting history was chosen because it also improved commit-message quality.

ABQ, an open-source universal test runner, distributes test suites across parallel CI jobs through a queue so workers can pull tests dynamically and finish more evenly than with fixed file-based partitions. Its initial design used a clustered model in which one worker generated the test manifest and reported results while all workers synchronized their exit statuses, but this created costly idle time when slow tests delayed completion and made flaky-test retries inefficient because entire clusters had to be rerun. Built-in automated retries reduced some failures but did not eliminate the need for full-suite reruns. ABQ was redesigned around independent “atomic” workers that exit when their assigned tests are complete, retain records of dispatched tests so individual workers can be retried with the same workload, and use a separate reporting command to aggregate results after the cluster finishes. The team released the revised tool publicly and plans further work on compute-efficient handling of flaky CI tests, alongside its related Captain product.

Captain is an open-source CLI designed to reduce engineering productivity losses caused by flaky tests, inefficient parallel test execution, and fragmented failure reporting across CI builds. Compatible with 15 testing frameworks, it can automatically retry failed tests under configurable limits, identify and manage known flaky tests, and quarantine unreliable tests so they continue running without failing a build. Captain can also split test files into balanced parallel partitions using historical execution times, while producing aggregate Markdown failure summaries that simplify reviewing results across multiple CI jobs. Its optional Cloud subscription adds automatic flaky-test detection, continuously updated timing data, web-based quarantining, analytics, and simpler configuration management, while the open-source version supports repository-managed configuration and timing files.

ABQ is a generally available open-source universal test runner that distributes test suites across CI workers through a queue-based model, aiming to avoid the uneven workloads and maintenance demands associated with manually or historically pre-partitioned test files. Rather than replacing existing frameworks, it wraps native test commands and currently supports bindings for RSpec, Jest, and pytest, allowing teams to retain their established tooling. Its features include automatic retries for failed tests, aggregated failure reporting, asynchronous network operations, multiple processes per machine, persisted dispatch history, and universal result formats such as JUnit XML and RWX JSON. ABQ can be deployed through a self-hosted binary and queue or managed queues provided by RWX, with installation and integration intended to be quick for existing CI workflows.