September 2026 Summaries

SpeedrunBench is a benchmark for evaluating LLM game-playing agents by the number of frames required to reach specific goals in 10 retro and open-source games across multiple genres, extending beyond conventional completion-based metrics that can saturate once games are solved. It assesses agents in online play, where they select actions from the current frame, and offline optimization, where they iteratively edit and replay full action traces either from a supplied successful route or from scratch. These settings are intended to measure multimodal understanding, exploration, route discovery, and long-horizon coherence, while paired commercial and open-source games allow researchers to examine possible pretraining contamination. Early results show substantial gaps between agents and human speedrunning records, including Pokémon Blue runs that remain roughly four to nearly 12 times slower than the human first-badge record. The findings also suggest that seed trajectories can constrain stronger models by anchoring them to inefficient routes, although they help weaker models complete otherwise difficult tasks. Increasing the number of runs per optimization iteration appears especially influential, with GLM 5.2 coming within 58 frames of the Super Mario Land 1-1 human record when given up to 200 runs per iteration.

SpeedrunBench is a benchmark for evaluating LLM agents across 10 retro and open-source games by measuring the number of frames required to reach defined goal states, extending beyond conventional completion-based metrics that can quickly saturate. It tests agents in online play, where actions are predicted frame by frame, and two offline settings: one that optimizes an existing successful trajectory and another that requires constructing a route from scratch. These settings assess multimodal perception, exploration, route discovery, and long-horizon consistency, while paired commercial and open-source games help examine potential pretraining contamination. Results show substantial gaps between agent performance and human speedrunning records, with the best Pokémon Blue first-badge runs about four times slower than the human record, while seeded trajectories can sometimes constrain stronger models by anchoring them to inefficient routes. However, seeds remain important for weaker models that cannot reliably complete games independently, and the number of evaluations per iteration appears especially influential, as GLM 5.2 came within 58 frames of the Super Mario Land 1-1 human record when allowed up to 200 runs per iteration.