Repository Intelligence: Building the Next Generation of Agent Evaluation Data

The text discusses the challenges and methodologies in evaluating software engineering agents, highlighting the inadequacies of traditional benchmarks like HumanEval when tested on real-world repository-level tasks. The research underscores the importance of Repository Intelligence, where agents must understand complex code dependencies and maintain state across long-running tasks. It introduces SWE-Bench+ and USEbench as more rigorous frameworks for evaluating agents' effectiveness in navigating these complexities. Potpie's approach involves creating a dynamic data pipeline and synthetic benchmarks to test agents on three cognitive functions: QA for deep dependency reasoning, CodeGen for context-aware synthesis, and Debugger for fault localization. The text critiques the limitations of open datasets, which often fail due to data contamination, lack of execution-based verification, and inadequate context. It describes the process of selecting diverse open-source repositories and outlines the technical methodologies employed to automate testing data creation. The evaluation suite's design aims to move beyond static knowledge and basic code generation to test agents' abilities to perform genuine reasoning, integration fidelity, and debugging in high-entropy environments. The text concludes by emphasizing a future roadmap focused on self-play reinforcement learning to create an evolving evaluation framework that adapts to agent capabilities.