The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents

ClickHouse has released data-agent-mnist, an open, reusable benchmark harness for evaluating agentic analytics models against a company’s own data warehouse, using 201 anonymized and synthetically reconstructed questions drawn from production traffic of its internal DWAINE assistant. Unlike conventional text-to-SQL tests, the benchmark evaluates agents that must discover schemas, plan across multiple steps, execute queries, and return correct result sets, with ground truth established through agreement among models from Anthropic, OpenAI, and Google and answers judged by a provider-diverse panel. In tests of 28 models, Claude Fable 5.1 achieved the highest correctness score at 76.6%, while several open-weight frontier models ranked highly and DeepSeek V4 Flash offered a much cheaper option, scoring 65.7% for roughly $1 per full evaluation compared with $52 for Fable 5.1. The findings suggest that model performance depends strongly on the specific warehouse and workload, that discovering relevant tables is often more difficult than writing executable SQL, and that incorrect planning accounts for most failures. The project also includes privacy safeguards, contamination testing, deterministic data generation, and scripts intended to let organizations create repeatable, tailored benchmarks and compare future models under consistent conditions.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.