SimpleQA and SimpleQA Verified explained: what they measure and how search APIs compare

SimpleQA is a factuality benchmark introduced by OpenAI in 2024 containing 4,326 short, stable questions with single verifiable answers, designed to assess both a language model’s factual knowledge and its ability to abstain when uncertain. Its grading labels responses as correct, incorrect, or not attempted, with an F-score combining answer accuracy and precision among attempted answers, although the metric can reward aggressive guessing. Google DeepMind’s 2025 SimpleQA Verified reduced the set to 1,000 questions by removing duplicate and biased items, reconciling source conflicts, balancing topics and answer types, clarifying numeric tolerances, and improving automated grading. Both benchmarks are intended primarily for measuring knowledge stored in models without external tools; when web search is enabled, performance becomes near-perfect because the questions generally concern easily retrievable, well-documented facts. In one vendor-reported September 2026 comparison of search APIs, systems scored between 91% and 97% on a 100-question sample, suggesting the benchmark is more useful for checking basic retrieval reliability and comparing cost than for distinguishing advanced research capabilities. The text also notes risks from public benchmark contamination, limited applicability to long-form accuracy, and the importance of consistent graders, larger samples, cost reporting, and filters preventing agents from accessing benchmark answer keys.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.