Building a Code Review system that uses prod data to predict bugs

Sentry's AI Code Review system, part of the Seer AI debugger, leverages Sentry data to accurately predict bugs in code changes, aiming to reduce noise by focusing on real issues rather than false positives. The system uses a multi-step pipeline involving hypothesis generation and verification to provide bug predictions. It employs filtering to focus on error-prone files, particularly in large pull requests, and analyzes the code using various tools and historical data to draft and verify bug hypotheses. The review process integrates production context from Sentry, using past issues and runtime data to improve the accuracy of bug predictions. The system also includes an evaluation mechanism to ensure high-quality predictions by assessing precision, recall, and accuracy, with improvements over time through regular testing and dataset collection. Additionally, context mocking is used during evaluations to maintain consistency and avoid performance issues. The system is continuously evolving, with ongoing efforts to enhance its contextual understanding and make its predictions more actionable for users.