ReviewBench: An open benchmark for AI code review

ReviewBench is a public offline benchmark for evaluating AI code review agents, designed to address shortcomings in existing evaluations by combining realistic pull requests, broad ground truth, configurable scoring, and reproducible validation. Its corpus contains 219 pull requests from 187 open-source repositories across 19 languages, with distributions modeled on analysis of 103.9 million GitHub pull requests and an emphasis on substantive multi-file changes. Findings are collected from human reviews, follow-up commits, static analysis, and frontier language models, then deduplicated and validated using a published rubric and an LLM grader; independent senior-engineer review achieved 96.6% agreement with the benchmark’s labels. ReviewBench measures grounded and augmented precision, recall, and F1 scores, allowing systems to receive credit for newly discovered valid findings while retaining a consistent basis for comparisons, and results can be filtered by severity, category, and precision-versus-recall preferences. GitHub reports that its internal use of the benchmark for Copilot code review has generally predicted the direction of later production experiments, including a multi-model review test that improved measured precision, recall, comment volume, critical findings, and cost efficiency. The research preview provides the dataset, methodology, leaderboard, and self-service tooling for developers to assess and submit their own agents.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.