May 2024 Summaries

Amazon Web Services (AWS) is renowned for its robust commitment to uptime, exemplified by its Service Level Agreements (SLAs) that often guarantee 99.9% uptime, with some services like DynamoDB Global Tables and AWS Key Management Service even promising 99.999% uptime. This commitment is underpinned by a sophisticated automated deployment pipeline that minimizes downtime through mechanisms such as automated rollbacks, progressive rollouts, and extensive monitoring of both service metrics and their dependencies. AWS's deployment strategy includes implementing "one-box" and "waves" stages to ensure service stability and compatibility during updates, allowing for controlled and gradual deployment across Availability Zones and regions. The process incorporates "bake" periods to detect regressions, extending deployment timeframes to ensure reliability but allowing for faster rollouts by monitoring traffic and metrics. Not just application code, but also feature flags and configuration changes are deployed through these pipelines, enhancing both safety and velocity. While companies like Graphite are inspired by AWS's processes, they face challenges in adopting these practices due to differences in scale and speed requirements, although they recognize the value of automation and are gradually integrating AWS-inspired strategies, such as blocking deployments during non-business hours and considering one-box deployments.

In 2005, Linus Torvalds faced a pivotal moment when the Linux kernel team had to find an alternative to the proprietary version control system, BitKeeper, after a licensing conflict arose. Originally, Linus himself managed kernel changes manually before BitKeeper introduced a distributed system that alleviated his workload. Despite its efficiency, BitKeeper's restrictive license eventually led to a clash when a developer reverse-engineered it, prompting Torvalds to seek alternatives. This urgency led to the creation of Git, which was initially a simple content-addressable file system rather than a full-fledged version control system. Git's design, which embraced distributed version control, was driven by Linus's dissatisfaction with centralized systems like CVS and SVN. Although Git was born out of necessity to continue Linux development, it eventually revolutionized source control management globally. The story of Git's inception underscores the complex interplay of technical challenges, licensing issues, and community collaboration, illustrating how innovation often arises not from sudden inspiration but rather from practical necessity and iterative refinement.

Graphite has introduced Graphite for Startups, a program designed to make its software development tools more accessible to early-stage companies and open source projects by offering them the full Standard plan features for free, including unlimited stacking, insights, automations, and a stack-aware merge queue. This initiative targets startups with 10 or fewer GitHub organization members and aims to enhance their development processes by reducing time spent blocked on code review, thereby accelerating their ability to iterate and scale. Additionally, Graphite for Open Source extends similar benefits to maintainers of open source projects, supporting the broader OSS community by providing them with advanced tooling. The program underscores Graphite's commitment to empowering small and agile teams to grow efficiently and potentially outgrow the program as they scale.

The recent launch week for Graphite introduced several new features aimed at enhancing developer collaboration and efficiency. Key announcements included the ability for multiple engineers to collaborate seamlessly on the same stack of pull requests (PRs) with automatic resolution of complex Git states. Additionally, Graphite now supports automatic rebasing of partially merged stacks, ensuring accurate diffs and correct reviewer assignments. The introduction of Graphite merge rules offers flexible branch protections and customizable code ownership, allowing teams to enforce specific requirements while maintaining compliance with company policies. A new batching workflow groups PRs for combined CI testing, which can reduce costs and expedite merging processes. Furthermore, integrations with Buildkite and GitHub Actions were introduced to help optimize CI pipelines, keeping releases reliable and costs manageable.

Graphite is addressing the challenge of growing continuous integration (CI) costs faced by rapidly-scaling companies by introducing the Graphite CI optimizer, designed to work with Buildkite and GitHub Actions. This solution helps manage CI pipelines for stacked pull requests (PRs) by running CI selectively on certain PRs in a stack, such as the bottom or top PRs, to control costs and maintain release reliability. The optimizer can configure different test suites, allowing faster tests to run consistently and slower, more comprehensive tests to run only when necessary, such as before merging. Batch merging in the Graphite merge queue further reduces unnecessary CI runs by optimizing when and how tests are conducted. The CI optimizer integrates with Buildkite and GitHub Actions by adding specific steps or jobs to pipelines, allowing for efficient and controlled CI execution. Currently in private beta with some enterprise customers, Graphite plans to expand access to a public beta soon.

Almost three years after transitioning from Facebook to a nascent startup with only five employees, the writer reflects on the unexpected learning experience and challenges faced in a smaller, less structured environment. Initially, fears revolved around a lack of technical support and potential catastrophic failures, yet the reality proved different as they discovered that the learning curve at an early-stage startup is more manageable than anticipated. The challenges of attracting and retaining users were significant, contrasting starkly with the vast user base of big tech, while early adopters offered invaluable feedback and support. Technical issues like a problematic database query provided real-time learning experiences, yet these challenges arose later than expected, allowing for gradual expertise development. Ultimately, the journey highlighted that, despite the inherent difficulties of startup life, an eagerness to learn can equip one to handle technical challenges more adeptly than initially feared.

Graphite has introduced a new batch merging feature for its merge queue system, designed to enhance throughput and reduce Continuous Integration (CI) costs for teams using the platform. This feature allows multiple pull requests (PRs) to be grouped together into a single temporary PR, running CI on the batch and merging it collectively if the tests pass, thus optimizing the merge process without slowing down releases or inflating CI costs. Batch merging is effective for managing a higher volume of PRs and can be customized for each repository's desired throughput. Unlike the Parallel CI strategy that ensures every individual PR passes CI, batch merging trades off the need for every single commit to build correctly for faster queue times and reduced CI expenses, while still ensuring that all tests pass at the head of the trunk branch post-merge. In case of a CI failure, options include removing the entire batch or bisecting to identify the problematic PR, with bisecting functionality being developed. Currently, the batch merging feature is available in public beta and can be enabled in merge queue settings.

Google employs two internal code review tools, Critique and Gerrit, to manage and streamline their code review process. Critique, primarily used internally, features a customizable dashboard to track in-flight changes, while Gerrit, open-sourced for public projects like Chromium and Android, provides a detailed overview of change lists. Both tools offer unique functionalities such as test coverage metrics, commit message as a reviewable entity, and visual differentiation for moved code lines. A distinctive aspect of Google's code review is the "attention set," which indicates whose action is needed to progress a change list. To ensure high standards, changes must pass three levels of approval: LGTM, code owners, and a Google-specific readability review. Google's rigorous review process emphasizes not only functionality but also adherence to coding standards and best practices, contributing to a culture where code review is prioritized and thoroughness is valued, resulting in a median review time significantly shorter than industry averages.

Graphite has introduced a highly-requested feature that automatically rebases stacked pull requests (PRs) after partial merges, streamlining the process for developers. Previously, partially merging PRs required manual intervention to rebase the remaining stacks, which could lead to inaccurate code diffs and workflow issues on GitHub. With this update, Graphite eliminates the need for additional commands like "gt sync && gt submit," ensuring that teammates see the correct diffs and the right reviewers are assigned. The new functionality integrates seamlessly into existing workflows without requiring new commands or tools. When a stack is partially merged, Graphite automatically rebases the remote branches of the unmerged PRs, maintaining the correct base and preventing any incorrect code diffs. This enhancement builds upon the previous introduction of shared stacks, allowing developers to experience a more seamless collaboration with Graphite handling updates to remote branches, akin to working with a Graphite bot. The feature has been rolled out to all organizations, reflecting Graphite's goal of improving development workflows without disrupting existing practices.

Graphite has introduced a new feature that allows developers to collaborate on stacked pull requests (PRs) with their teammates in real-time, addressing a long-standing request from users. This functionality simplifies the complex process of managing collaborative stacks of PRs from multiple authors, which is challenging with standard Git. The feature, available from version 1.3.4 of the Graphite CLI, enables users to fetch, create, and submit changes using commands like `gt get`, `gt create`, and `gt submit`, thereby facilitating seamless teamwork where different team members can work on various parts of a feature simultaneously. Graphite's system automatically resolves the complex Git states that arise during collaboration, ensuring that when local changes are detected as more recent than remote ones, it rebases the local state to incorporate incoming changes without manual intervention. This advancement in collaborative stacking supports efficient division of labor and enhances the overall code review process, allowing teams to build complex features more effectively.

Many large technology companies develop custom-built tools as they scale, driven by the desire for more control and seamless integration with existing systems. For instance, Google and Meta have historically created in-house solutions like Critique and Gerrit for code reviews instead of relying on platforms like GitHub, which is more commonly used by companies like Airbnb and Stripe. The tendency for companies to build their own tools—often termed "not invented here" syndrome—is debated, with arguments for maintaining control over features and integrations versus leveraging the flexibility and cost-effectiveness of external SaaS tools. The decision to develop in-house solutions may be influenced by the size of the company and the era in which it was founded, as newer companies might be more inclined to adopt external tools due to the broader availability of SaaS products. This raises questions about whether the preference for custom-built tools is an inevitable consequence of scale or a result of the company's foundational culture and era.