safety security
No Open Model Cleared 42 Percent on Unhinted Security Commands
KaliBench tests 8,504 intent-to-command pairs across 1,642 Kali Linux tools.
Summary
KaliBench tests 8,504 intent-to-command pairs across 1,642 Kali Linux tools.
KaliBench measures whether models can translate analyst intent into executable command lines, with deterministic canonicalization, alias-aware scoring, sandbox execution and human refinement. Across 24 configurations, no open-weight model exceeded 42 percent exact-command accuracy without tool hints. Fine-tuning and verifiable-reward training substantially improved an eight-billion-parameter model to performance comparable with a tested 685-billion-parameter mixture-of-experts model. Exact-command accuracy is a benchmark metric, not authorization to run generated security commands unsupervised.
Why it matters
KaliBench tests 8,504 intent-to-command pairs across 1,642 Kali Linux tools.
Limits and context
- Exact-command accuracy is a benchmark metric, not authorization to run generated security commands unsupervised.
Key claims
KaliBench tests 8,504 intent-to-command pairs across 1,642 Kali Linux tools.
Qualification: Exact-command accuracy is a benchmark metric, not authorization to run generated security commands unsupervised.
Evidence: source-2026-10-02-005
Sources
- arXiv preprint 2610.02206arXiv · primary research
Corrections
No corrections have been recorded for this story.