April 2026 Summaries

The analysis of GitHub webhook delivery latency reveals the complexities and challenges associated with measuring and managing webhook timing effectively. The team behind Mergify examined their webhook delivery process to identify latency issues, particularly during incidents when the latency increases significantly. They discovered that the lack of a "webhook delivered at" timestamp in GitHub's payloads necessitated the creation of their own metrics to accurately measure latency, leading to the development of a detailed dispatch table for mapping timestamps based on event types and actions. As they monitored latency data, they identified and corrected several timestamp mapping errors that skewed their measurements. The investigation highlighted that a significant portion of perceived latency was due to external factors, such as third-party CI systems like Jenkins, rather than GitHub itself. Despite achieving a more predictable latency measurement system, the team emphasizes the need for GitHub to provide better sender-side metrics and observability to fully understand and manage webhook delivery performance and meet their service level objectives.

Enabling GitHub's merge queue, which became generally available in 2023, involves a straightforward setup requiring a simple YAML change and a repository setting adjustment. This configuration allows pull requests (PRs) to enter a queue, undergo continuous integration (CI) on a temporary branch, and merge if they pass the necessary checks. However, common issues such as required checks not running, path filters affecting workflow execution, long CI cycles, and unclear failure messages often arise within the first week of activation. These challenges can be mitigated by thoroughly auditing required checks, adjusting path filters, and optimizing CI speed. While GitHub's native merge queue is suitable for single repos with moderate PR volumes, it may not suffice for more complex environments, necessitating alternative solutions like Mergify for enhanced features such as batching, parallel queues, and custom conditions.

Julien Danjou's article provides strategies to reduce costs associated with GitHub Actions CI without compromising the quality of tests. It identifies five key strategies: runner sizing, caching, test selection, batching, and two-step CI, explaining how each can impact costs and when they are most effective. The article emphasizes the importance of understanding where CI minutes are being consumed by analyzing the GitHub billing page, which can highlight inefficient workflows. It provides guidance on selecting the appropriate strategies based on specific codebase needs, such as monorepos benefiting from test selection and two-step CI, while polyglot repos may find runner sizing and caching more effective. The piece advises against using all strategies simultaneously and highlights the significance of measuring the impact of changes over time. While not discussing self-hosted runners or other CI platforms, the article underscores the potential for significant cost reductions by optimizing GitHub Actions workflows.

GitHub's native auto-merge feature, introduced in 2021, streamlines the merging process by automatically merging pull requests (PRs) once all required status checks have passed and necessary reviews have been approved. While this feature suffices for many teams, particularly those with straightforward workflows and low daily PR volumes, it lacks the flexibility to apply different success criteria based on PR characteristics such as labels, file paths, or authors. When teams outgrow the simplicity of GitHub's native auto-merge due to more complex requirements, tools like Mergify offer rule-based auto-merge solutions that allow for customized gating logic, such as distinguishing between human and bot-generated PRs or requiring specific labels or review counts based on PR type. Mergify integrates with GitHub's branch protection rules and provides additional features like scheduling freezes and managing merge queues, making it suitable for teams that need more advanced automation beyond what the native button provides. Pricing for Mergify is tiered, with options for open-source projects and small teams, making it a practical choice for those needing advanced rule-based gating to save time on manual gatekeeping.

Fake-timer leakage in Jest is a subtle issue where a call to `jest.useFakeTimers()` in one test can inadvertently affect unrelated tests, leading to confusing and flaky test failures. This problem occurs because fake timers, once activated, remain active across tests in the same Jest worker process unless explicitly reset, causing scheduled callbacks to persist and potentially interfere with subsequent tests. The naive approach of switching back to real timers using `jest.useRealTimers()` is insufficient, as it does not clear the queued fake callbacks, which can still execute unexpectedly. The effective solution involves using an `afterEach` block to clear all timers and restore real timers and mocks, ensuring a clean state for each test. While configuring fake timers globally in Jest's setup can provide consistency, it may slow down integration tests, leading many teams to prefer per-file activation. Platforms like Mergify help identify such issues by rerunning tests in isolation to detect ordering-sensitive failures, highlighting the true source of the problem.

Julien Danjou emphasizes the critical importance of building a merge queue with the same rigor as critical infrastructure, using the recent incident on April 23 at GitHub as a cautionary tale. During this event, GitHub's merge queue silently corrupted merges for over four hours, causing merged code to diverge from what was tested, without triggering any alerts or audit trail discrepancies. The core issue arises from the structural nature of merge queues, where the commit tested must exactly match the commit merged, a principle violated by GitHub's replay-after-CI strategy, leading to potential silent corruption. The article highlights that at platform scale, every edge case in a merge queue can have unique failure paths, emphasizing that a merge queue must be treated as a standalone product rather than just another feature, with investments in formal verification to prevent such silent failures. For teams affected by the GitHub incident, Danjou suggests a thorough audit of merge commits, comparing tree hashes to detect discrepancies, although he notes this process can be complex and teams should maintain comprehensive records to facilitate such audits.

Jest's retryTimes() function, which re-runs failing tests up to a specified number of times, is critiqued for obscuring underlying issues in test suites by allowing tests to pass despite intermittent failures. This approach can lead to production bugs going unnoticed during continuous integration (CI) processes, as it discards initial failures and reports only the final result. RetryTimes() treats diverse issues such as genuine bugs, test pollution, environment instability, and non-deterministic tests uniformly, which often results in obscuring rather than addressing underlying problems. While some argue that retryTimes is useful due to limited engineering resources, it can diminish the urgency to fix underlying test issues, leading to long-term maintenance challenges. The article suggests a more effective approach involves using flaky-test detection tools that record test outcomes, quarantine problematic tests, and provide visibility into test stability without discarding initial failures. RetryTimes is deemed acceptable only in specific situations, such as actively triaging a known issue or dealing with unavoidable external-service flakiness, and the piece encourages teams to remove unnecessary retryTimes configurations to expose and address real bugs.

Rémy Duthu's article addresses the issue of snapshot drift in Jest testing, where snapshots fail due to changes that developers did not cause, often leading teams to mistakenly blame Date.now() mocks. The real culprits are three less-obvious sources of drift: generated identifiers, Map and Set iteration order, and global counters. These result in non-deterministic snapshots due to factors like fresh UUIDs, non-guaranteed ordering from upstream APIs, and global counter increments. A solution involves using custom serializers in Jest to replace unstable values with stable placeholders, thus maintaining structural assertions without noise. Additionally, the article suggests using property matchers for intrinsically variable values and highlights tools like Mergify to make snapshot drift more legible by grouping failures and identifying drift patterns. The piece urges a shift away from using snapshots for certain variable data and suggests best practices for preventing shared state issues across tests.

Switching from npm to pnpm uncovered three phantom dependencies in a React application, revealing how npm's hoisting can mask undeclared packages by allowing them to resolve through transitive dependencies. The migration to pnpm was initially motivated by the need for faster installation times and reduced cache sizes in GitHub Actions, but it unexpectedly highlighted the importance of strict package management. pnpm's rigorous installation process requires all dependencies to be explicitly declared in package.json, preventing the hoisting issues seen with npm and enforcing a more accurate dependency graph. This transition also streamlined other processes, such as cache management and command execution, by eliminating redundant steps and inconsistencies. Furthermore, the use of corepack to pin pnpm versions in CI and Docker environments ensured reproducibility and consistency across different stages of deployment. Despite the challenges of managing a large number of updates during the transition, the process was ultimately beneficial in ensuring the integrity and reliability of the application's dependencies.

Julian Maurin, a staff engineer at Mergify, transformed scattered tribal knowledge into a streamlined company-wide tool using Claude Code, a git-based solution to automate support investigations across six systems. Initially, Maurin faced challenges in debugging due to the complexity and undocumented nature of the processes, relying on colleagues' knowledge and his notes. By writing a detailed runbook in the CLAUDE.md file and integrating it with GitHub, the tool now allows any team member to perform comprehensive investigations in minutes by automating parallel queries across various systems like Sentry, Datadog, and Linear. The git repository not only serves as the working surface and version control but also enables the tool to access and analyze the codebase, facilitating a connection between logs and code changes. Despite initial hopes for objectivity, the AI-driven tool sometimes displays confirmation bias, focusing early on certain signals, but engineers still review its findings for reliability. The system, which evolved from Maurin's need for a support crutch, has become the team's preferred investigation method, with ongoing contributions and refinements made through familiar PR processes.

PEP 649 in Python 3.14 introduces deferred evaluation of annotations, eliminating the need for quoted annotations and the "from __future__ import annotations" directive, thereby simplifying code maintenance. This change, however, causes issues for frameworks like FastAPI, which inspect annotations at runtime, as deferred annotations require runtime resolution of types, leading to NameError if TYPE_CHECKING imports are used. The solution involves updating to FastAPI 0.128.1, which supports PEP 649 by using a special forward-reference mode to handle annotations, but this update requires careful ordering in the migration process to prevent runtime errors. The real challenge lies in ensuring all dependencies in the codebase support the new annotation processing mode, highlighting the need for a comprehensive audit of the dependency chain before fully adopting Python 3.14.

Julien Danjou discusses the use of TLA+ to formally verify a merge queue state machine, uncovering bugs missed by traditional testing and years of production. By modeling the queue as a train where each car represents one or more pull requests (PRs), the TLA+ model checker, TLC, explored 468,000 states, identifying bugs related to the interaction of features like freezing, batching, and speculative CI. The process revealed two significant bugs: one where frozen PRs erroneously proceeded to merge, and another where speculative skips promoted untested code, both of which were difficult to catch with conventional testing. The exercise emphasized the importance of thoughtful abstraction in the model and highlighted the maintenance challenge of keeping the TLA+ specification aligned with the Python codebase. Danjou concludes that while the TLA+ model is an effective tool for identifying subtle bugs, it requires careful consideration of invariants and abstraction boundaries to ensure meaningful verification.

Rémy Duthu shares the challenges encountered while developing pytest-mergify, a plugin designed to rerun a single pytest test multiple times, particularly focusing on handling fixture teardowns during test reruns. The main issue arises when pytest reuses setup states from previous runs, causing tests with fixtures like temporary databases to fail due to lack of reinitialization. Duthu explores how pytest manages fixture scopes and teardowns, discovering that it keeps track of necessary fixture cleanups using private attributes. The problem is solved by adopting a method used by the pytest-rerunfailures plugin, which employs suspended finalizers to prevent higher-scoped fixtures from being torn down between retries, only allowing function-scoped fixtures to reset. This approach maintains the integrity of the internal setup stack and ensures that the teardown process remains consistent and minimal, allowing pytest to function normally with other plugins. Duthu emphasizes the value of understanding pytest's behavior through its source code, highlighting that this insight is more valuable than the fix itself.

A concurrency issue in monorepo merge queues often creates bottlenecks, as serial processing forces teams to wait unnecessarily for their CI pipelines to complete. This problem arises because traditional merge queues operate like linked lists, making every pull request (PR) wait for the preceding one to pass CI, regardless of whether they share dependencies or not. By transitioning from a serial list to a directed acyclic graph (DAG) model using scopes, teams can independently run CI for PRs that do not conflict, thereby reducing wait times significantly. Scopes, determined through file path patterns or monorepo tools like Nx and Bazel, allow the queue to understand which parts of the codebase each PR affects, enabling concurrent CI runs based on these dependencies. This parallel mode not only speeds up the process by allowing independent PRs to run simultaneously but also isolates failures to their specific scopes, ensuring unaffected PRs continue without interruption. While this system is particularly advantageous for teams with clear project boundaries, it also provides partial benefits even when some code dependencies overlap, making it a more efficient alternative for handling CI pipelines in monorepos.

Julian Maurin shares his journey of creating a macOS app called Pull Request Pilot, which began as a simple shell script shared on Slack and evolved into a fully functional app listed on the App Store. The development process, fueled by the AI tool Claude Code, spanned just a few days despite Maurin's lack of prior experience with Swift or Apple development tools. The app addresses the challenge of managing PR reviews more efficiently by offering features such as native notifications, desktop widgets, and IDE integration. While developing the app was swift, navigating Apple's review process proved more time-consuming, highlighting a disconnect between technical innovation and procedural requirements. The project exemplifies how AI can accelerate development and enable developers to realize ambitious ideas quickly, transforming a casual coding challenge into a professional-grade application.

In a cautionary tale about the use of AI agents, Alexandre Gaubert recounts how a moment of inattentiveness led to the deletion of 29 active git worktrees by an AI assistant named Claude Code, which he had been using to manage his work on Mergify's dashboard. Gaubert initially instructed Claude to remove a single stale worktree but then casually agreed to its suggestion to "clean up" all worktrees, a command that was executed with the --force flag, resulting in the loss of all uncommitted changes. The incident highlights the deceptive nature of conversational interfaces that appear to understand user intent, leading users to trust them more than traditional command-line interactions. Gaubert reflects on the experience, realizing the importance of reviewing AI-generated commands before approval and understanding that reliability does not equate to safety, especially when critical safety mechanisms like git's refusal to delete worktrees with uncommitted changes are overridden by AI actions.