Q2D-Web: Evaluating first-stage retrievers at scale

Q2D-Web (Query2Doc-Web) is a private benchmark and public leaderboard for assessing retrieval models used in agentic retrieval-augmented generation systems at web scale. It includes approximately 190 million deduplicated web documents and 69,721 privacy-filtered, agent-reformulated queries spanning ten languages, sampled from nine months of production search traffic and covering diverse domains. To address sparse and potentially biased relevance labels, it offers citation-based, production web-ranking, and expanded LLM-judged relevance sets, averaging 88.5 positive judgments per query in the combined set. The benchmark emphasizes corpus size, query volume, and depth of relevance annotations as necessary for realistic evaluation, particularly because smaller corpora can remove difficult distractors and overstate retrieval quality. An RRF-based subcorpus containing 31.7% of documents was designed to reduce evaluation costs while largely preserving full-corpus model rankings. Results use Recall@1000 as the primary metric, with pplx-embed-v1-4b achieving the highest Recall@1000 across all judgment sets, while Nemotron-3-Embed-8B led on certain higher-ranking metrics; the report also notes that Perplexity models could have an in-distribution advantage despite exclusions from training data. Publicly available Hugging Face models can be submitted for standardized evaluation under specified implementation and access requirements.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.