benchmarks evals
Passing Once Collapsed to 25 Percent Across Twenty Runs
Thinkingbox evaluates terminal backend state, policy compliance and collateral effects across 507 business workflows.
Summary
Thinkingbox evaluates terminal backend state, policy compliance and collateral effects across 507 business workflows.
The strongest tested model reached 65.36 percent pass-at-one but only 25.25 percent success across twenty attempts on the new benchmark. Many failed runs terminated cleanly after valid state-changing actions, supporting the authors’ argument that a plausible response or tool call is not evidence that the right persistent state transition occurred without extra effects.
Why it matters
Thinkingbox evaluates terminal backend state, policy compliance and collateral effects across 507 business workflows.
Limits and context
- The strongest tested model reached 65.36 percent pass-at-one but only 25.25 percent success across twenty attempts on the new benchmark.
- Many failed runs terminated cleanly after valid state-changing actions, supporting the authors’ argument that a plausible response or tool call is not evidence that the right persistent state transition occurred without extra effects.
Key claims
Thinkingbox evaluates terminal backend state, policy compliance and collateral effects across 507 business workflows.
Qualification: The strongest tested model reached 65.36 percent pass-at-one but only 25.25 percent success across twenty attempts on the new benchmark.
Evidence: source-2026-08-23-007
Sources
- arXiv preprint 2608.19741arXiv · primary research
Corrections
No corrections have been recorded for this story.