November 2023 Summaries

Continuous Integration and Continuous Delivery (CI/CD) systems are essential for software development, with GitHub Actions, Jenkins, and others forming the backbone of many organizations' software deployment processes. However, optimizing CI/CD pipelines to balance speed, cost, and efficiency is a common challenge, particularly in the face of time and financial costs that can hinder development. Graphite addresses these issues by offering strategies such as parallelization, caching, selective testing, and implementing the fail-fast principle to streamline CI/CD workflows. Parallelization aids in maintaining fast CI runtimes, while caching and artifact management help avoid redundant work. Selective testing targets only changed components to improve efficiency, and the fail-fast principle prevents wasting resources on failing processes. The guide further suggests that understanding the team's development workflows, such as the choice between monorepos and polyrepos, can significantly affect CI/CD management. These strategies, which Graphite has successfully implemented, are presented as adaptable solutions for different organizational contexts to enhance CI/CD pipeline performance while keeping costs in check.

Graphite transitioned from a simple single-page application (SPA) deployment to a robust containerized environment using AWS ECS to better manage, deploy, and scale their Next.js application. This shift was driven by the need for enhanced scalability, performance, and cost efficiency as their platform and user base grew, moving away from the limitations of their previous SPA setup hosted on S3. After considering various hosting options, including Vercel and AWS Amplify, Graphite chose ECS for its alignment with their AWS VPC preferences, cost-effectiveness, and the ability to unify frontend and backend hosting strategies. This decision allowed them to implement sophisticated deployment strategies, such as blue-green deployments, while maintaining performance and security within their AWS infrastructure. The migration involved adopting best practices such as optimizing for cold starts, automating infrastructure with Terraform, and managing costs effectively, ensuring their ECS-hosted applications remain robust and scalable. Through this journey, Graphite emphasizes the importance of evaluating infrastructure needs, embracing containerization, investing in automation, and staying flexible to adapt to new technologies.

Managing and minimizing complexity in software engineering is crucial, as overly complex codebases can lead to "change amplification," where simple changes require widespread modifications, slowing down development. This issue, common in early web development, has been mitigated by modern practices that centralize design elements. An analysis of 1.5 million pull requests (PRs) reveals that those involving fewer file changes merge faster, highlighting the need for systems that minimize file touch points to maintain high engineering efficiency. Surprisingly, as PR complexity increases, the time spent reviewing each file decreases, possibly due to reviewers adopting broader evaluation strategies and encountering more repetitive changes. This data underscores the importance of keeping PRs small to ensure efficient reviews and high-quality code, with strategies like "stacking" and simplifying software design being recommended to manage complexity. Tools like Graphite facilitate these practices, helping teams maintain development velocity and reduce the risk of regressions by limiting change amplification.

Over a six-week period, significant improvements were made to the performance of Graphite's application server, nicknamed "subwoofer," by addressing key issues that caused intermittent latency spikes and server inefficiencies. The investigation revealed that Graphite's P95 latency metrics were sometimes misleading due to outliers and integration with third-party APIs like GitHub, leading to unnecessary scaling and troubleshooting. By delving deeper into endpoint-specific latency spikes and event loop blockers, the team discovered that excessive synchronous computation and large database queries were major culprits, prompting a shift to worker threads and more efficient query handling. Additionally, the realization that not all background processes impact top-line performance led to strategic optimizations, such as unshipping non-essential features. These efforts resulted in smoother, more consistent performance metrics, allowing for better detection of regressions and a clearer understanding of overall site performance trends. The ongoing work focuses on reducing endpoint latencies and sustaining the improvements achieved, providing a more reliable user experience.

Trunk-based development (TBD) is a workflow where developers work off a single, shared main branch, integrating their changes frequently, which contrasts with the feature branch workflow that involves isolated, long-lived branches leading to siloed work and potential merge conflicts. TBD is increasingly favored by organizations aiming for high-velocity software development due to its benefits such as enhanced team synchronization, fewer merge conflicts, and streamlined continuous integration and delivery processes. This approach promotes regular commits, ensures the codebase remains current, and facilitates faster feedback loops, which are critical for maintaining a deployable state. Industry giants like Google and Facebook have long employed TBD, and data indicates that it leads to higher development speed, with a significant reduction in pull request open-to-merge time compared to other methods. As of 2023, the trend toward TBD continues to grow, with nearly 80% of pull requests being based on main branches, and it is advocated as a best practice for organizations looking to improve software development efficiency and quality.

Git, an essential tool for developers, was created by Linus Torvalds in 2005 as a response to a licensing dispute with BitKeeper, and it quickly became the most widely used version control system, with adoption rates soaring from 69% in 2017 to 94% in 2021. Unlike its predecessors, Git introduced a distributed model, allowing changes without needing a remote server, and it improved the software development workflow with features like branch merging and commit tagging. The system organizes data into four object types: blobs, trees, commits, and tags, which are processed through three key areas: the working area, the staging area, and the repository. Git commands are divided into "porcelain" for user interface and "plumbing" for internal functions, enabling developers to manage their code efficiently. Understanding these internals and operations helps developers utilize Git more confidently, enhancing their workflow by making processes faster and more reliable.

Developers frequently face the choice between merging and rebasing when integrating changes from one branch into another in a git repository. While both approaches have their merits, the trend among fast-moving companies and large repositories is increasingly favoring a rebase-centric workflow, particularly in the context of trunk-based development. This approach is seen as beneficial for creating a cleaner, more understandable history by avoiding the clutter of merge commits and maintaining a linear trunk branch. Rebasing offers advantages such as easier reversion of commits and avoidance of the complexities associated with long-lived branches, which are less common in closed-source development environments. However, rebasing can be intimidating due to the complexity of Git commands and the frequent need for force pushes, which clutter the GitHub PR timeline. Tools like the Graphite CLI and platforms like Facebook's Phabricator help automate and simplify the process, promoting the adoption of a squash-rebase-merge workflow. Despite some challenges, the shift toward rebasing is evident among large tech companies and newer repositories, driven by the desire for efficient and clean code integration practices.

Continuous Integration (CI) is crucial in modern software development, primarily by automating checks on Pull Requests to free engineers for more nuanced tasks, yet the duration of CI processes has become a subject of debate. While a 10-minute CI cycle is often cited as ideal, Graphite's data-driven analysis reveals that CI times between 15-30 minutes may correlate with higher productivity in terms of merged Pull Requests, suggesting that slightly longer CI times might not necessarily hinder development efficiency. The study found that longer CI times increase the time to merge due to repeated cycles, which can accumulate to significant delays, and highlights that CI times exceeding 30 minutes may require optimization. The concept of stacking workflows, where changes are broken into smaller, parallel PRs, is recommended to alleviate CI delays, maintaining development velocity despite lengthy CI runs. The data suggests a nuanced understanding of CI times, advocating for a balance between speed and complexity, with an emphasis on adapting CI practices to optimize productivity effectively.