TheMachine Press

A daily newspaper for the age of artificial intelligence.

Morning editionPermanent story

safety security

The Safety Guard Notices the Wrapper

A three-level audit finds that command length, tool names and formatting markers can change both guard decisions and reported benchmark scores.

Published Updated Story ID: mp-2026-10-17-002
Read the complete editionStory JSON

Summary

A three-level audit finds that command length, tool names and formatting markers can change both guard decisions and reported benchmark scores.

A safety benchmark can accidentally reward a guard for recognizing the way a risky action is written instead of the risk itself. The Shortcut Sensitivity study separates three questions: whether labels can be predicted from superficial cues, whether guard decisions correlate with those cues and whether controlled rewrites actually change a decision. In LinuxArena, shorter average shell commands strongly predicted trajectory labels. Under domain holdout, an exact-tool lookup baseline on AgentDojo-Traj fell from an F1 of 0.901 to 0.495. On a fixed safe-row cohort, rewriting an information marker lowered blocking by 8.33 percentage points, while replacing tool names with opaque aliases raised it by 11.46 points; all unsafe calls in that screen remained blocked. The paired effects appeared for the tested Haiku configuration but not Luna or deterministic Q8 TS-Guard. The result is not a universal ranking of guards. It is evidence that representation shortcuts can alter both behavior and the score meant to measure safety.

Why it matters

A three-level audit finds that command length, tool names and formatting markers can change both guard decisions and reported benchmark scores.

Limits and context

  • The paired effects appeared for the tested Haiku configuration but not Luna or deterministic Q8 TS-Guard.
  • The result is not a universal ranking of guards.

Key claims

  1. A three-level audit finds that command length, tool names and formatting markers can change both guard decisions and reported benchmark scores.

    Qualification: The paired effects appeared for the tested Haiku configuration but not Luna or deterministic Q8 TS-Guard.

    Evidence: source-2026-10-17-002

Sources

  1. arXiv preprint 2610.22243, original v1arXiv · primary research

Corrections

No corrections have been recorded for this story.