Behind SWE-rebench: Infrastructure to collect massive datasets of SWE tasks and evaluate agents at scale

Software engineering agents, powered by large language models (LLMs), have rapidly advanced, yet the technical challenges of large-scale experimentation persist due to the need for distributed orchestration beyond single-machine capacity. Nebius’ AI R&D team has focused on building scalable infrastructure to support such experiments, involving the creation of extensive datasets and evaluation pipelines like SWE-bench and SWE-rebench. Their work necessitated using distributed systems such as Kubernetes and TractoAI to manage the orchestration of thousands of agent runs and evaluations, while overcoming challenges related to data processing, container management, and task execution. By leveraging Kubernetes for flexible agent orchestration and TractoAI for efficient data processing and evaluation, the team developed a robust framework that allows for efficient experimentation and sharing with the research community. This infrastructure is designed to facilitate automated and reliable performance measurement of SWE agents, which operate by executing code within containers, much like a human engineer, but with the added complexity of requiring a scalable and distributed backend to handle the vast amounts of data and workload demands. Nebius is now opening this infrastructure to the broader research community, including offering support through their research credits program, to accelerate progress in the field of software engineering agents.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.