How we investigated prompt cache misses with Datadog

Datadog engineers investigated why internal AI applications were not consistently benefiting from LLM prompt caching despite having it enabled, using Cloud Cost Management to identify provider spending and Agent Observability to trace individual model requests, token usage, latency, and cache reads and writes. They found that caching requires requests to share an identical beginning, so unstable details such as timestamps placed early in system prompts, nondeterministic ordering of tools or files from Go maps, missing provider-specific cache controls, and use of unsupported API compatibility routes could all prevent reuse. The team recommended placing stable instructions before changing context, sorting generated tool and file lists, verifying that cache settings reach provider request bodies, and using supported APIs where appropriate. They also stressed that improvements should be evaluated across complete tasks rather than isolated calls, accounting for cache-write fees, output and retry costs, response time, task accuracy, and provider billing data to confirm actual savings.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.