safety security
The Assistant Knows the Time and Still Misses the Constraint
TASC finds models often calculate elapsed time correctly without applying the safety rule that depends on it.
Summary
TASC finds models often calculate elapsed time correctly without applying the safety rule that depends on it.
The benchmark contains 3,418 conversational inputs derived from 128 templates plus 33 scenarios that separate time calculation from action. Across 13 models, assistants account for time-dependent constraints in 30.5 percent of cases where the requested action violates a rule, while seldom warning unnecessarily. In 62.1 percent of failures, the model had calculated the elapsed time correctly. Supplying timing explicitly fixes only about half. TASC is a benchmark of templated everyday constraints, but it exposes a practical gap between temporal reasoning and advice.
Why it matters
TASC finds models often calculate elapsed time correctly without applying the safety rule that depends on it.
Limits and context
- Supplying timing explicitly fixes only about half.
Key claims
TASC finds models often calculate elapsed time correctly without applying the safety rule that depends on it.
Qualification: Supplying timing explicitly fixes only about half.
Evidence: source-2026-11-05-016
Sources
- arXiv preprint 2611.06839, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.