Content Deep Dive

StateBench: Testing how language models track state changes over long documents

Blog post from Hugging Face

Post Details
Company
Date Published
Author
Kyle Wiggers
Word Count
1,628
Company Posts That Month
60
Language
-
Hacker News Points
-
Post removed?
No
Summary

StateBench is a benchmark developed by AI2 to measure how well language models track changing information about entities across long documents, a capability relevant to code execution, storytelling, and multi-step reasoning. It organizes state-tracking tasks into four increasing levels of complexity, from fixed assignments and simple overwrites to cumulative updates and reference-based variable changes, drawing on distinctions from Krohn–Rhodes theory. Tests of pretrained open models found that most handled simple fixed-value tasks but declined on reassignment tasks and performed near chance on the most complex general finite problems, with results also varying substantially by presentation format. Frontier models including Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna performed strongly on simpler levels but deteriorated sharply on longer sequences involving chained or reference-based updates. When trained directly on StateBench, recurrent Gated DeltaNet and hybrid architectures generalized effectively to much longer sequences on most task types, while transformers still failed to learn the hardest category, suggesting architectural limitations. The researchers note unresolved questions about constrained Gated DeltaNet variants, numerical precision, attention combinations, and tracking newly introduced entities, and release StateBench to support more detailed evaluation and development of robust long-context memory systems.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
Gemini 3.8 Flash 4 281 36 21 +37%
Opus 5.5 2 1,032 79 41 +65%
LLM 1 3,355 742 191 -49%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.