An (actually useful) framework for evaluating AI code review tools

Benchmarks, while traditionally seen as objective measures, often reflect the biases and limitations of their creators, leading to potential manipulation and misrepresentation, as seen historically with database performance benchmarks and now with AI code review benchmarks. The text advocates for a personalized approach to evaluating AI code review tools, emphasizing the importance of using one's own benchmarks tailored to specific organizational needs, codebases, and standards. It suggests designing a representative evaluation dataset, defining ground truth and severity levels, and selecting metrics that truly inform decision-making, such as detection quality and developer experience. The recommendation is to combine controlled offline benchmarks with in-the-wild pilot testing to assess a tool's real-world effectiveness. Emphasizing coverage and configurability over narrow precision, the text warns against relying solely on vendor-defined benchmarks, which can often serve more as marketing tools than accurate reflections of performance.