March 2024 Summaries

Managing code reviews and ownership in monorepos is complex, with the common use of GitHub's CODEOWNERS file proving insufficient due to its single-file constraint that doesn't cater well to multiple organizational needs. The monorepo build tool, Bazel, is mentioned as related but not ideal for handling such "whole-repo operations" because of its dependency graph limitations. Alternatives like rules_codeowners attempt to layer ownership on top of Bazel's graph but introduce inefficiencies by requiring large registration blocks that slow down builds due to unnecessary dependency fetching. Google's approach with a separate OWNERS file and Kubernetes' similar method are highlighted as more effective, as they maintain ownership semantics without involving Bazel. A more flexible solution is proposed, where ownership changes are treated as a continuous delivery problem, allowing updates to OWNERS files to be merged and reflected in CODEOWNERS only after they are committed to the main branch, ensuring that required reviews are conducted before ownership modifications take effect.

Google implemented an /experimental folder in their monorepo to facilitate code experimentation without maintaining long-term commitments to the code. This setup allows developers to use shared library code and receive continuous integration feedback while collaborating with others. However, incorporating experimental code into the main branch poses risks, such as unintended dependencies in production services. To mitigate this, Google uses Bazel's testonly feature to restrict dependencies on experimental code, ensuring it is only used for testing purposes. The buildozer tool is employed to automate the configuration of BUILD files under the /experimental folder, marking them as testonly. Additionally, a continuous integration task is set up to query and report any non-testonly targets in the /experimental directory, preventing misuse. Developers are also provided with commands to resolve any CI failures by updating BUILD files accordingly.

In this detailed account, the author explains the process and challenges faced while adapting a pattern for publishing Go binaries from the bazel-lib repository to the rules_py repository, which involved transitioning from Go to Rust for tool development. The shift was motivated by the desire to leverage open-source work in the Python ecosystem but encountered difficulties due to differences in cross-compilation support between Go and Rust. The solution involved using GitHub Actions to automate the building of binaries, integrating integrity hashes into the release artifacts, and modifying the release process to be more streamlined and less reliant on manual maintenance. Key features of the release process include the exclusion of non-essential files to prevent bloating, stamping release tags into source files, and ensuring that pre-built binary toolchains are fetched efficiently for users. The process is designed to be reproducible and maintainable, ensuring that releases can be easily managed by a small team. The author emphasizes the importance of a streamlined process for ensuring a seamless end-user experience and mentions future updates as the rules_py project approaches its 1.0 release.

The text discusses an approach to manage the fetching of NLTK data for Machine Learning tasks under Bazel, highlighting the drawbacks of the traditional installation method suggested by NLTK, which involves non-hermetic and non-reproducible processes that rely on machine state. Instead of using the typical installation path that requires network access and manual setup, the author suggests using Bazel's capabilities to download and manage the data more efficiently and reproducibly. By employing the Bazel Downloader and the http_archive helper, the required NLTK data, such as the Punkt tokenizer, is fetched directly from GitHub, ensuring that it is available in a consistent manner across different environments. The process involves setting up a cache folder structure that mimics NLTK's expectations, using a macro to facilitate documentation and organization, and configuring the Python target to use the downloaded data by setting the NLTK_DATA environment variable. This method ensures that the data is hermetically provided to the test action at runtime without needing network access, thus enhancing portability and reliability.