Evaluating Agents Against What They Are Actually Supposed to Do [Testμ 2026]

Francesca Lazzeri’s Testμ Conf 2026 keynote argued that generic AI benchmarks can miss critical agent failures, such as policy violations, improper refunds, unsafe tool use, or ignored approval boundaries, even when agents score well on broad measures like helpfulness and groundedness. She presented Microsoft’s open-source ASSERT framework, or Adaptive Specification-driven Scoring for Evaluation and Regression Testing, which converts plain-language behavioral requirements into taxonomies, generated test cases, and policy-based evaluations for models and agents. The proposed evaluation process combines written specifications, common baseline measures such as groundedness, retrieval quality, relevance, fluency, and safety, agent-specific metrics such as task adherence and tool-call accuracy, and production observability that turns real failures into new regression tests. Using a customer-support assistant as an example, the session emphasized evaluating the agent’s full trajectory rather than only its final response, while monitoring live quality, repeat contacts, latency, escalations, and cost per resolution. Lazzeri also highlighted tokenomics and controlled or causal experimentation as ways to connect agent quality improvements to operational costs, revenue, and measurable business outcomes.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.