Which Bugs AI Agents Fix Better With Traffic

A benchmark of 100 hand-authored bugs in an unfamiliar 240-service codebase found that an AI coding agent fixed 55% of cases using production alerts alone, compared with 77% when given captured failing requests and responses. Captured traffic was especially effective for bugs whose evidence appears in network payloads, raising success rates for race and write-path issues from 22% to 89%, state-machine transitions from 72% to 92%, streaming and multipart framing from 52% to 85%, and cross-service contract drift from 44% to 81%, while deep framework-internal bugs improved only marginally from 75% to 79%. A service map improved results by six percentage points, substantially less than the 28-point gain from traffic, suggesting that payload details such as field names help agents locate relevant code more effectively than topology alone. The experiment used roughly 18,000 model calls at an estimated cost of $118, aided by extensive token caching, but its author notes that results are directional for smaller bug categories and that one-shot fixes still require verification. The proposed next step is to replay captured traffic against a running service so agents can test and validate their patches before opening pull requests.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.