benchmarks evals
Show the Updated Appointment and Success Goes From 0 to 15
EnvOS certifies that mid-task changes fired, the task stayed solvable and the grader is correct before it blames a computer-use agent.
Summary
EnvOS certifies that mid-task changes fired, the task stayed solvable and the grader is correct before it blames a computer-use agent.
A changing appointment, policy or record creates several possible failure sources: the agent may miss it, the update may not fire, or the grader may be wrong. EnvOS certifies each change before a run and re-checks the verdict from archived evidence afterward. In clinic booking, Claude Opus 5 succeeded on 0 of 16 tasks with no extra cue, 2 of 16 after a reminder to re-check and 15 of 16 when the updated value was shown. In claim approval, Opus 5 re-checked on its own and scored 10 of 10, while Sonnet 5 rose from 1 of 10 to 9 of 10 when shown the current policy. On blind corrupted episodes, the pipeline accepted none of 26 as clean. These are small, constructed settings, but they isolate interface effects from agent and grader failures.
Why it matters
EnvOS certifies that mid-task changes fired, the task stayed solvable and the grader is correct before it blames a computer-use agent.
Limits and context
- A changing appointment, policy or record creates several possible failure sources: the agent may miss it, the update may not fire, or the grader may be wrong.
Key claims
EnvOS certifies that mid-task changes fired, the task stayed solvable and the grader is correct before it blames a computer-use agent.
Qualification: A changing appointment, policy or record creates several possible failure sources: the agent may miss it, the update may not fire, or the grader may be wrong.
Evidence: source-2026-10-30-007
Sources
- arXiv preprint 2610.40246, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.