May 2022 Summaries

The author discusses how they converted their Node.js library to Deno using a runtime adapter pattern. They highlight the key differences between Node.js and Deno, such as TypeScript support, module resolution, standard libraries, and built-in globals. To adapt their library for Deno, they rewrote import paths, swapped out imports from the Node.js standard library with Deno equivalents, and injected Node.js globals into the Deno-ified code. They also wrote a script using Deno to automate these changes. The author suggests that this approach can be useful for other library authors looking to support Deno.

Deno is a modern JavaScript runtime designed to natively support TypeScript and address some of the shortcomings of Node.js, as conceived by its creator, Ryan Dahl. The text outlines the process undertaken by EdgeDB to adapt their Node.js client library to be compatible with Deno, thereby eliminating the need for maintaining separate codebases. This adaptation leverages a "runtime adapter" pattern to bridge differences between the two environments, such as TypeScript execution, module resolution, and standard library usage. The conversion process involves rewriting import paths to align with Deno's URL-based module system and replacing Node.js standard library imports with their Deno equivalents. Furthermore, a codemod script is developed to automate these changes and handle Node.js global variables, facilitating seamless integration of the library with Deno. The process is automated within a continuous integration workflow to maintain a Deno-compatible version of the library, which is released to a separate repository.

EdgeDB, a company that uses GitHub Actions for its CI workflow consisting of thousands of tests and various build processes, has managed to reduce the runtime from 2+ hours to just 10 minutes by sharding their test suite across several workers. They achieved this by evenly splitting up their 5000-ish tests into separate "shards" using GitHub's build matrix feature. The algorithm they used for sharding takes into account the setup conundrum and ensures that no single slow-poke delays the whole run. This approach allowed them to distribute the workload across multiple workers, significantly speeding up their CI process.

EdgeDB significantly optimized its continuous integration (CI) workflow by utilizing GitHub Actions to parallelize a comprehensive suite of tests, achieving a runtime reduction from over two hours to just 10 minutes. The test suite, which consists of approximately 5000 tests and 60 database setups, was split into shards using a custom algorithm that allocates tests into evenly distributed buckets to maximize efficiency. The process involves creating priority queues for both standalone and setup-linked tests, and strategically allocating them to ensure minimal setup initialization and efficient resource use. By employing a distributed computing approach, each runner independently executes the sharding algorithm, thereby avoiding the complexities of having a single control arbiter. EdgeDB's solution includes a final integrity check to ensure all tests are executed and uses a CSV file for tracking test runtimes, which is updated after each CI run. This innovative use of GitHub Actions demonstrates its versatility for managing complex workloads efficiently, resulting in a CI process that is approximately 95% faster.