LLM Cost Tracking for Agent Evals: Gate on Cost per Success

LLM agent evaluations should track cost per verified successful task alongside pass rate, calculating total agent spending across all attempts—including failures, retries, tool loops, and sub-agents—divided by successes supported by observed evidence rather than self-reports. The proposed approach tags every model call with an evaluation case and role, records token usage with a run-pinned price table, separates model-judge costs from agent costs, and uses repeated runs with high-percentile budgets to account for variability. A Pareto-style release gate then compares a candidate against a baseline on both pass rate and cost per verified success, rejecting changes that are more expensive without meaningful quality gains while requiring explicit decisions for accuracy-cost tradeoffs. Research cited in the text argues that accuracy-only benchmarks can reward costly retrying and complex agent designs without improving outcomes, while existing tools such as promptfoo, Langfuse, DeepEval, and Braintrust can record cost but generally require custom logic to enforce this combined metric. For evidence-based verification, Agent Assurance and its Rook CLI are presented as tools that assess observed effects such as file changes, records, and tool calls, helping ensure that the successes used in cost calculations are genuinely verified.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.