Cutting our inference bill by 68% by being obsessive about optimization

Cloudflare reduced the average cost of its AI code reviews from about $2.43 to $0.77 and median completion time from 4 minutes 9 seconds to 1 minute 32 seconds by optimizing prompt caching, reducing unnecessary model calls, eliminating unused tools and redundant follow-up steps, skipping irrelevant reviewers on repeat reviews, and cancelling jobs that no longer needed to run. A major issue was that repository-specific information prevented shared prompt prefixes from being reused across sessions, causing costly cache writes despite high cache-hit rates; restructuring prompts and coordinating shared prefixes substantially lowered spending. Additional savings came from testing newer models and effort settings, which reduced token use, although lower effort sometimes diminished review quality. Evaluation using author code changes and blinded model assessments indicated that code-quality reviews at high effort retained roughly the same number of meaningful findings while producing less noise and fewer overstated severities, whereas performance reviews lost some important findings at medium effort and security results remained less certain. Reliability also improved, with fewer reviews ending without a verdict and far fewer security-review retries, while the company plans to test higher effort settings for performance and security reviewers.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.