The Agentic Analytics Benchmark, round two: GPT-6, Claude Opus 5.5, Claude Sonnet 5.5

ClickHouse’s follow-up Agentic Analytics Benchmark evaluates five newly added models—three GPT-6 variants and Claude Opus and Sonnet 5.5—on 201 SQL analytics questions over the same synthetic warehouse, 60-turn limit, and independent judge panel used previously. Claude Fable 5.1 remains the leader at 76.6%, while all new models scored in a statistically overlapping 68.7% to 69.7% range, placing them in the middle of the 34-model board and below GPT-5.5 at default reasoning effort. Claude Sonnet 5.5 showed the largest improvement over its predecessor, raising pass rate by 6.5 points, eliminating turn-limit failures, improving speed and cost efficiency, and performing notably better on questions involving a dimensional CRM layer. Across the additions, incorrect planning remained the dominant failure mode, while SQL runtime errors were absent, suggesting that deciding what analysis to perform is a greater challenge than query execution. The benchmark harness was also updated with environment pinning, credential checks, configurable numeric tolerances, improved result-column matching, uncertainty modeling for judges, experiment hooks, and a LibreChat runner, while contamination tests found no evidence that the new models had memorized the benchmark questions. The authors emphasize that aggregate leaderboard results conceal important trade-offs in cost, latency, and completion behavior, and recommend that organizations run similar benchmarks using their own schemas and questions.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.