safety security
The Safety Test Simulates the Tools, Not Just the Attacker
On LinuxArena, simulated control evaluations predicted five monitors' safety with 3.6 percentage points of error at a 5 percent audit budget.
Summary
On LinuxArena, simulated control evaluations predicted five monitors' safety with 3.6 percentage points of error at a 5 percent audit budget.
Control evaluations often test an adversarial model in a purpose-built environment, then extrapolate to deployment. Simulated control evaluations replace real tool responses with model-generated ones tailored to the deployment task. Against executable LinuxArena ground truth, they predicted safety across five monitors with 3.6 percentage points of root-mean-square error at a 5 percent audit budget, versus 10.1 points for the separate BashArena setting. They also reproduced attack and monitor behavior more closely and used fewer than 5 percent as many task-pair draws. The result is evidence for the reported environments, not proof that simulation faithfully covers every deployment.
Why it matters
On LinuxArena, simulated control evaluations predicted five monitors' safety with 3.6 percentage points of error at a 5 percent audit budget.
Limits and context
- The result is evidence for the reported environments, not proof that simulation faithfully covers every deployment.
Key claims
On LinuxArena, simulated control evaluations predicted five monitors' safety with 3.6 percentage points of error at a 5 percent audit budget.
Qualification: The result is evidence for the reported environments, not proof that simulation faithfully covers every deployment.
Evidence: source-2026-10-23-015
Sources
- arXiv preprint 2610.31593, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.