benchmarks evals
The Research Agent Inherits the Messy Workspace
ARE-bench preserves unfinished files, old decisions and failed attempts instead of evaluating agents in a clean room.
Summary
ARE-bench preserves unfinished files, old decisions and failed attempts instead of evaluating agents in a clean room.
ARE-bench contains 75 tasks reconstructed from real researcher sessions and full in-progress workspaces across seven research stages. Its 1,173 binary checklist items include not only the immediate deliverable but also historical decisions and material workspace constraints. Across seven frontier models, the strongest passed every item on 66.2% of tasks. Three models passed about 69% when judged only on delivery contracts and general norms, but 28.4% to 40.0% when historical and workspace requirements were added, showing how clean-task evaluation can overstate reliability.
Why it matters
ARE-bench preserves unfinished files, old decisions and failed attempts instead of evaluating agents in a clean room.
Limits and context
- Its 1,173 binary checklist items include not only the immediate deliverable but also historical decisions and material workspace constraints.
- Three models passed about 69% when judged only on delivery contracts and general norms, but 28.4% to 40.0% when historical and workspace requirements were added, showing how clean-task evaluation can overstate reliability.
Key claims
ARE-bench preserves unfinished files, old decisions and failed attempts instead of evaluating agents in a clean room.
Qualification: Its 1,173 binary checklist items include not only the immediate deliverable but also historical decisions and material workspace constraints.
Evidence: source-2026-11-11-013
Sources
- arXiv:2611.14364v1arXiv / Cornell University · primary research
Corrections
No corrections have been recorded for this story.