benchmarks evals
The World Model Loses Cells as the History Grows
Grid-Bench executes every instruction, making the full final state—and each wrong cell—directly checkable.
Summary
Grid-Bench executes every instruction, making the full final state—and each wrong cell—directly checkable.
Grid-Bench represents two-, three- and four-dimensional discrete worlds, applies eight families of operations and asks a language model to predict the complete state after a textual event history. Because a reference executor produces the answer, evaluation can measure both exact recovery and how many cells are wrong. Across six models, complete-state accuracy fell sharply with longer histories, and systems with similar exact-match scores sometimes differed substantially in partial correctness. Some models still recovered changes beyond copying the initial grid. The benchmark is synthetic by design, but it isolates compositional state tracking without relying on a subjective judge.
Why it matters
Grid-Bench executes every instruction, making the full final state—and each wrong cell—directly checkable.
Limits and context
No additional limitation was separately recorded.
Key claims
Grid-Bench executes every instruction, making the full final state—and each wrong cell—directly checkable.
Evidence: source-2026-10-18-015
Sources
- arXiv preprint 2610.22312, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.