20× the CI traffic without getting slower: How we rebuilt Git serving at Datadog

Datadog developed gitretriever, a distributed Git mirroring system designed to reduce CI bottlenecks caused by expensive repository fetches, particularly from large monorepos and increasing traffic from automated services and AI coding agents. Rather than scaling its replicated GitLab backend, which increased replication overhead, the company separated a small set of GitHub-synchronized mirrors from an autoscaled fleet of relays that distribute repository updates without consensus or repeated upstream fetches. Gitretriever reduces work through branch-level synchronization, content-addressed packfile replication, and a cache for identical fetch requests, while a read-only API lets many workloads retrieve specific Git information without cloning entire repositories. Following a gradual, feature-flagged rollout with automatic fallback, the service handled more than a billion requests across roughly 5,500 repositories, grew traffic about 20-fold in four months while maintaining median latency near 40 milliseconds, and reduced fetch-serving CPU on the prior backend by three to four times. The project was built by two engineers using Claude Code alongside detailed design documentation, integration testing, metrics, and logs, with future plans including wider rollout, expanded API capabilities, and repository copies directly on CI nodes.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.