December 2023 Summaries

Code reviews are an essential component of efficient engineering processes, aiming to catch defects early, distribute context among team members, and maintain code quality as codebases grow. To achieve this, a balance between speed and accuracy is crucial, with lightweight processes enabling rapid feedback while maintaining standards. Techniques for improving code reviews include starting with a clear description, using smaller pull requests, and stacking related PRs for parallel workstreams, which can enhance efficiency and reduce development delays. To maintain manageable scopes, reviews should be limited to under 60 minutes and 400 lines of code to ensure focus and thoroughness, while avoiding context switching can further streamline the process. Checklists and automation can standardize and simplify reviews, allowing developers to concentrate on judgment-based tasks rather than rote processes. A predictable review cadence and short review cycles can enhance team velocity, while metrics tracking can offer insights for continuous improvement. Employing tools like Graphite facilitates the stacking process, promoting small, focused pull requests that expedite reviews and foster best practices with minimal training.

As traffic and database demand are expected to grow tenfold, a Node.js web app team anticipates the need to scale their SQL query handling from hundreds of millions to billions daily, requiring pinpointing slow queries to prevent performance bottlenecks. The team initially considered using Datadog for tracing but found it cost-prohibitive and ineffective for their background worker-heavy load. They also attempted monkey patching, which proved impractical. Ultimately, they implemented call site tagging, a solution where SQL queries are tagged with user-defined strings indicating their origin in the codebase, improving traceability, legibility, and cost-effectiveness. Despite challenges with the TypeORM library, they achieved nearly 100% coverage by wrapping all TypeORM calls in a query builder API, adding tagging, and using linting rules to ensure compliance. This approach allowed them to quickly identify and optimize problem areas, preparing them for anticipated user growth and database scaling, while also considering further improvements like supporting tagging across all TypeORM APIs and building internal tools to prevent untagged queries.

Git reflog is a valuable tool for software engineers, providing a detailed chronological log of changes made to the HEAD pointer in a Git repository, which can help recover from mistakes like accidental commits or deletions. It allows users to navigate the granular history of their repository, effectively acting as a time machine by enabling them to rewind and undo unwanted actions. The reflog is stored in the .git directory and can be filtered using time qualifiers to explore specific points in history, making it easier to identify and correct errors. Engineers can utilize commands like `git reflog expire` to manage the size of the reflog by pruning older or unreachable entries, while `git reflog delete` allows for individual entry removal. A scenario illustrating the use of reflog involves correcting a faulty rebase by identifying the commit before the rebase and resetting the branch to that state. Despite its capabilities, reflog is not a substitute for good practices, and engineers are encouraged to maintain safe Git habits to minimize the need for its use.

On December 6, 2023, Graphite's web application experienced a brief outage lasting 124 seconds, affecting the pull request functionality and causing minor disruptions for nearly a dozen users who encountered 404 errors. The incident was detected by CTO Greg Foster after a user report on Slack, and the engineering team, including Alyssa and Brendan, quickly identified and resolved the issue by addressing a misconfiguration in the S3 bucket settings, restoring the site to normal functionality. The root cause was traced back to an unintended change in the S3 production bucket configuration, which was promptly reverted. In response, Graphite plans to implement new policies, such as testing configuration changes in a staging environment before deploying to production, and adding linter warnings to prevent jinxing statements about system reliability. The incident report, provided by Foster, highlights the swift response and collaboration within the team and the supportive role of the community during the service disruption.

The text explores the challenges of traditional GitHub pull request (PR) workflows, highlighting issues such as large PR sizes, slow reviews, and reduced code quality, which can lead to bottlenecks in the development process. It introduces stacking as a solution, a development approach that breaks down large features into small, dependent PRs, facilitating quicker and more effective reviews. Stacking, supported by tools like Graphite, allows developers to incrementally integrate changes, reducing the risk of bugs and technical debt while promoting continuous delivery. This method enhances productivity by enabling concurrent progress, simplifying bug fixes, and accelerating review cycles, ultimately streamlining collaboration and improving the overall development workflow.