Beyond evals: A/B test AI features with LLM traces and business metrics

GrowthBook has introduced beta support for using LLM trace data as metrics in controlled online experiments, helping teams evaluate model, prompt, agent, and parameter changes with real users before broad releases. By connecting trace information such as token costs, latency, errors, tool calls, and evaluation scores with product metrics including revenue, feature adoption, activation, retention, and user feedback, teams can assess both AI quality and business impact. The platform assigns users to experiment variations, tags their traces with the assigned version, retrieves those traces from integrations such as Langfuse and Arize Phoenix, and compares results using frequentist or Bayesian methods. GrowthBook provides default metrics including cost per user, p95 latency, error rate, token usage, calls per user, and traces per user, while allowing custom evaluation and feedback metrics. Suggested applications include testing cheaper models without sacrificing quality, improving assistant helpfulness, dynamically routing traffic among prompts, measuring whether larger models increase sales, and gradually rolling out agents with performance guardrails.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.