safety security
The Security Script Looks Plausible Until the Machine Checks It
HardEval executes model-written remediations on live virtual machines and grades the persistent system state with hidden compliance checks.
Summary
HardEval executes model-written remediations on live virtual machines and grades the persistent system state with hidden compliance checks.
Security hardening is not complete when a shell script looks reasonable. HardEval turns Red Hat Enterprise Linux 8 STIG controls into execution-verified tasks. A model receives the required outcome but not the implementation, writes a Bash script, and runs it on a live virtual machine; hidden OpenSCAP and OVAL checks determine whether the system actually moved from fail to pass. Across fourteen frontier and open-weight configurations, the authors found large and uneven differences among security domains. A recurring failure was convincing code that did not satisfy the exact compliance condition. The benchmark covers a specific operating system and control set, but it measures the durable state change that code-only evaluations miss.
Why it matters
HardEval executes model-written remediations on live virtual machines and grades the persistent system state with hidden compliance checks.
Limits and context
- Security hardening is not complete when a shell script looks reasonable.
- A model receives the required outcome but not the implementation, writes a Bash script, and runs it on a live virtual machine; hidden OpenSCAP and OVAL checks determine whether the system actually moved from fail to pass.
- A recurring failure was convincing code that did not satisfy the exact compliance condition.
Key claims
HardEval executes model-written remediations on live virtual machines and grades the persistent system state with hidden compliance checks.
Qualification: Security hardening is not complete when a shell script looks reasonable.
Evidence: source-2026-10-19-015
Sources
- arXiv preprint 2610.23685, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.