ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

ScarfBench is an open benchmark designed to evaluate AI agents' abilities in migrating enterprise Java applications across frameworks like Spring, Jakarta EE, and Quarkus, addressing a significant yet challenging area of software engineering. Unlike traditional benchmarks that focus on code generation, ScarfBench emphasizes not just code translation but also the preservation of application behavior, successful deployment, and dependency management, which are crucial for real-world applications. Despite advances in coding agents, the benchmark reveals that framework migration remains difficult, with current AI agents achieving low success rates in preserving application behavior. This difficulty is compounded by complex dependencies and environmental challenges, such as configuration and tool inconsistencies, highlighting the need for reliable validation and architectural reasoning. ScarfBench offers a comprehensive dataset, evaluation infrastructure, and a public leaderboard, serving as a valuable resource for researchers and practitioners aiming to improve AI-assisted application modernization while encouraging contributions of new migration scenarios and techniques.

No tracked trend matches for this post yet.