Introducing AutoEvals: Automatically find the best model for your task

AutoEvals is a new toolkit designed to help teams evaluate and select LLMs using their own recent production traffic rather than relying on broad public benchmarks. Integrated through an Inference Gateway or tracing SDK, it samples requests, replays them across candidate models, and uses LLM judges to assess output quality, behavioral similarity to the current model, cost, and latency. Its results include model recommendations, per-model summaries, and an interface for inspecting individual responses and judge reasoning, helping teams identify safe upgrades or lower-cost alternatives. The product’s creators report that internal testing revealed both that newer, cheaper models can outperform established options on specific agent tasks and that quality can converge among high-capability models, making additional spending unnecessary. AutoEvals is available to Inference account users free to start, aiming to make recurring model comparisons faster and less operationally burdensome.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.