What makes a good code review benchmark for AI tools?

As AI coding tools become more prevalent, the volume of code produced is increasing, leading to a gap between the speed of code generation and the capacity for human review. Qodo addresses this challenge by providing tools to help developers catch issues early and reduce reviewer load, maintaining high-quality development as speeds increase. The effectiveness of AI in code review is increasingly important, with benchmarks playing a critical role in evaluating AI systems' ability to handle real pull requests and provide useful feedback. However, there is currently no widely accepted benchmark for AI code review, as existing evaluations often focus on adjacent tasks rather than the full scope of code review. Qodo emphasizes the importance of building meaningful datasets from organic pull requests and synthetic data to create comprehensive benchmarks that reflect real-world development challenges. Establishing clear ground truth and focusing on both detection and resolution in benchmarks are crucial for meaningful evaluation. Qodo is working towards creating an industry-standard benchmark that combines organic and synthetic data, measures remediation quality, and ensures reproducibility to contribute to a foundation for effective evaluation of AI code review tools.