Code Review Bench: Towards Billion Dollar Benchmarks

Code Review Bench is a new open-source benchmark for code review tools, designed to address the limitations of static benchmarks like SWE-bench, which succumbed to Goodhart’s Law by becoming targets rather than measures of effectiveness. Unlike traditional benchmarks that rely on static datasets, Code Review Bench incorporates both offline controlled evaluations and online real-world developer behavior data, ensuring continuous updates and relevance. This approach allows the benchmark to self-correct when discrepancies arise between offline and online results, providing a more accurate reflection of a tool's utility. The initiative aims to create a dynamic foundation for measuring and improving code generation by using real-world developer interactions as a guide. This method contrasts with previous benchmarks that struggled with issues like dataset contamination and broken tests, as it continuously integrates new data and insights from actual developer feedback. The project is open-source and seeks collaboration from industry and academia to refine its methodology, aiming to bridge the gap between the high costs of training advanced AI models and the relatively low investment in maintaining accurate benchmarks.