Introducing SpeedrunBench: A Challenging Benchmark for Frontier Agents

SpeedrunBench is a benchmark for evaluating LLM game-playing agents by the number of frames required to reach specific goals in 10 retro and open-source games across multiple genres, extending beyond conventional completion-based metrics that can saturate once games are solved. It assesses agents in online play, where they select actions from the current frame, and offline optimization, where they iteratively edit and replay full action traces either from a supplied successful route or from scratch. These settings are intended to measure multimodal understanding, exploration, route discovery, and long-horizon coherence, while paired commercial and open-source games allow researchers to examine possible pretraining contamination. Early results show substantial gaps between agents and human speedrunning records, including Pokémon Blue runs that remain roughly four to nearly 12 times slower than the human first-badge record. The findings also suggest that seed trajectories can constrain stronger models by anchoring them to inefficient routes, although they help weaker models complete otherwise difficult tasks. Increasing the number of runs per optimization iteration appears especially influential, with GLM 5.2 coming within 58 frames of the Super Mario Land 1-1 human record when given up to 200 runs per iteration.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.