developer tools
The Debugger Looked at the Step Before the Click
Paired screenshots and action traces improved root-cause guidance for computer-use agent retries.
Summary
Paired screenshots and action traces improved root-cause guidance for computer-use agent retries.
CUADebug introduces a failure taxonomy, a human-annotated set of 204 failed OSWorld trajectories and a debugger that inspects suspicious before-and-after screenshots with action traces. Task reasoning and control accounted for 110 failures, more than perception, grounding or external-system categories. On the reported split, structured root-cause guidance roughly doubled continual re-execution success from 12.2 to 25.86 percent, still below human-oracle guidance at 29.21 percent. The evidence is benchmark-specific and leaves most failed tasks unresolved.
Why it matters
Paired screenshots and action traces improved root-cause guidance for computer-use agent retries.
Limits and context
No additional limitation was separately recorded.
Key claims
Paired screenshots and action traces improved root-cause guidance for computer-use agent retries.
Evidence: source-2026-08-05-010
Sources
- arXiv preprint 2608.02643arXiv · primary research
Corrections
No corrections have been recorded for this story.