Introducing GlobeBench: A Benchmark for Text-Based World Models

GlobeBench is an open-source benchmark for evaluating whether language models can accurately simulate environment responses after agent trajectories, containing 1,632 tasks spanning five domains, 19 simulation skills, and four capabilities: state coherence, causal correctness, interface fidelity, and constraint enforcement. Tasks provide a trajectory and final action, with model predictions compared against deterministic execution-derived reference states; 1,137 tasks were created by the authors and 495 adapted from existing executable backends. Across 20 general-purpose and specialized world models, GPT-6 Astra achieved the highest success rate at 49.57% (809 tasks), while no model exceeded 50%, 631 tasks were solved by none, and four Claude models incurred many failures through safety-related refusals. Performance was especially weak on schema conformance, long-context tasks whose decisive evidence occurred over 40 turns earlier, and actions with indirect effects on unnamed entities, where pooled success fell to roughly 5%. Incorrect predictions most often appeared as confident, well-structured outputs containing one or more wrong values, highlighting risks in using imperfect simulators for reinforcement-learning training. The authors note that the benchmark measures only single transitions rather than compounding rollout errors, and have released the tasks, answers, code, graders, model outputs, and judgments for reproducibility and further research.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.