The FRAMES benchmark explained: multi-hop retrieval for RAG and search agents

FRAMES (Factuality, Retrieval, And reasoning MEasurement Set) is a public Google benchmark introduced in 2024 for end-to-end evaluation of retrieval-augmented generation systems, containing 824 human-written questions that require combining facts from two to fifteen Wikipedia articles. It measures whether a system can retrieve relevant sources, extract information, and perform multi-hop reasoning—including numerical, temporal, tabular, constraint-based, and post-processing tasks—using final-answer accuracy scored by an LLM judge. The original study found that simple BM25 retrieval offered limited gains over no retrieval, while iterative search planning raised Gemini-Pro-1.5-0514 accuracy to 0.66, compared with 0.729 when all gold articles were supplied, indicating that both retrieval and reasoning remain important sources of error. Because it uses a fixed Wikipedia corpus, FRAMES is particularly suited to comparing RAG pipelines, retrieval strategies, rerankers, and agent loops, but it is less informative about live-web freshness, broad source coverage, or changes to Wikipedia since the benchmark’s creation. Systems can be evaluated by answering the dataset prompts, judging responses against gold answers, reporting overall and reasoning-type accuracy, and separately measuring recall of the annotated Wikipedia articles; comparisons should use consistent judge models and settings because historical scores and results from different evaluators are not directly interchangeable.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.