November 2025 Summaries

Vibe Code Bench is a benchmark designed to measure whether AI agents can build complete, deployable applications from a single natural-language specification, emphasizing “zero-to-one” software creation for non-programmers rather than isolated coding assistance. Its 100 application prompts draw on consumer products, startup concepts, and consulting case studies, while evaluation uses BrowserUse-based workflows and sub-steps to assess whether generated apps meet functional requirements, including front ends, back ends, databases, authentication, payments, and email services. The authors chose realistic natural-language specifications over rigid unit-test structures, customized an OpenHands-based agent harness, standardized tools and infrastructure such as Supabase, Stripe, and MailHog, and gave models a five-hour generation window with deployment guidance. Reported observations include compounded errors from early technical choices, unexpectedly solid functional performance but limited visual polish from some Chinese models, persistent instruction-following failures, the value of reviewing raw agent traces, and the importance of comparing models by cost and latency alongside accuracy. The project argues that evaluation methods must advance alongside increasingly complex AI generation, and proposes future extensions for repairing incomplete applications, handling iterative feedback, and testing broader technologies, with the broader aim of tracking a shift from AI coding copilots toward more autonomous software-building agents.