Pass@k vs Pass^k: Average Pass Rate Hides Flaky AI Agents

AI agent evaluations should distinguish average capability from consistency, because Mean@k or pass@1 can conceal tasks that succeed only intermittently, while pass@k measures whether at least one of several attempts succeeds and pass^k measures whether every attempt succeeds. IBM Research’s AppWorld results illustrate this gap: GPT-4.1 achieved a 77.4% average pass rate across five runs but passed all five runs on only 53.0% of tasks, with failures concentrated in a subset of flaky tasks despite temperature-zero settings. Pass@k is useful when outputs can be automatically verified and retried, such as code generation with unit tests, whereas pass^k is presented as more appropriate for customer-facing agents and release gates where each user receives one attempt. The text recommends estimating pass^k per task using an unbiased combinatorial formula based on repeated runs rather than raising an aggregate average to a power or using small-sample plug-in rates, and it stresses resetting environments and preserving realistic sources of variation. It also separates flaky behavior, in which unchanged tasks alternate between passing and failing, from regressions, in which formerly reliable tasks fail after a change; flaky tasks require repeated measurement and stabilization, while regressions warrant release blocks and investigation. For production use, it advises tracking per-task pass^k alongside averages and confidence intervals, using higher repeat counts for high-consequence scenarios, excluding unverifiable runs from pass rates, and retaining scenario snapshots and execution evidence to identify whether inconsistency originates in the agent, environment, or grader.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.