benchmarks evals
The Better Answer Arrives After the World Has Moved
Realtime-Bench separates reasoning quality from serving delay and finds that more thought can help or hurt when every second changes the task.
Summary
Realtime-Bench separates reasoning quality from serving delay and finds that more thought can help or hurt when every second changes the task.
An agent acting in a moving environment pays for every token with elapsed time. Realtime-Bench evaluates dynamic tasks in which delayed actions directly reduce success, rather than treating latency as a deployment detail outside the benchmark. Under its Wall Clock condition, where the environment advances during real inference, the best tested success rate reached 44 percent and the best Pass-to-the-third score reached 36 percent. A Paused Clock condition removes serving delay to expose the model's underlying task capability, while Token Clock advances simulated time according to generated token counts so temporal awareness can be studied apart from hardware speed. Across those views, the authors find that time costs—not only wrong decisions—block completion, and that higher reasoning effort has mixed effects under pressure. More deliberation can improve a choice while making the choice obsolete. The suite is a curated benchmark, not a complete model of robotics, games or live operations. Still, it supplies a missing measurement discipline: real-time agents must decide how much thinking the moment can afford, maintain perception while reasoning, and treat timely action as part of correctness rather than an optimization added later.
Why it matters
Realtime-Bench separates reasoning quality from serving delay and finds that more thought can help or hurt when every second changes the task.
Limits and context
- A Paused Clock condition removes serving delay to expose the model's underlying task capability, while Token Clock advances simulated time according to generated token counts so temporal awareness can be studied apart from hardware speed.
- Across those views, the authors find that time costs—not only wrong decisions—block completion, and that higher reasoning effort has mixed effects under pressure.
- The suite is a curated benchmark, not a complete model of robotics, games or live operations.
Key claims
Realtime-Bench separates reasoning quality from serving delay and finds that more thought can help or hurt when every second changes the task.
Qualification: A Paused Clock condition removes serving delay to expose the model's underlying task capability, while Token Clock advances simulated time according to generated token counts so temporal awareness can be studied apart from hardware speed.
Evidence: source-2026-10-18-002
Sources
- arXiv preprint 2610.22155, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.