Introducing SpeedrunBench: A Challenging Benchmark for Frontier Agents

SpeedrunBench is a benchmark for evaluating LLM agents across 10 retro and open-source games by measuring the number of frames required to reach defined goal states, extending beyond conventional completion-based metrics that can quickly saturate. It tests agents in online play, where actions are predicted frame by frame, and two offline settings: one that optimizes an existing successful trajectory and another that requires constructing a route from scratch. These settings assess multimodal perception, exploration, route discovery, and long-horizon consistency, while paired commercial and open-source games help examine potential pretraining contamination. Results show substantial gaps between agent performance and human speedrunning records, with the best Pokémon Blue first-badge runs about four times slower than the human record, while seeded trajectories can sometimes constrain stronger models by anchoring them to inefficient routes. However, seeds remain important for weaker models that cannot reliably complete games independently, and the number of evaluations per iteration appears especially influential, as GLM 5.2 came within 58 frames of the Super Mario Land 1-1 human record when allowed up to 200 runs per iteration.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.