developer tools
Every Model Tripped Over Cargo’s Carry Rule
SemVerBench found predictable version-constraint blind spots across npm, PEP 440 and Cargo—and a resolver solved the task.
Summary
SemVerBench found predictable version-constraint blind spots across npm, PEP 440 and Cargo—and a resolver solved the task.
SemVerBench contains 240 machine-checkable version-constraint questions with unique answers across npm, Python’s PEP 440 and Cargo. Six frontier models showed mechanism-specific failures: Cargo’s partial-comparator carry rule trapped every model near 60%, while a PEP 440 corner set split one model’s 0-of-26 result from another’s 97–100%. Supplying the governing rule recovered many errors; interval decomposition did not. Because a free resolver was effectively perfect, the authors recommend tool delegation over in-head version reasoning.
Why it matters
SemVerBench found predictable version-constraint blind spots across npm, PEP 440 and Cargo—and a resolver solved the task.
Limits and context
- Supplying the governing rule recovered many errors; interval decomposition did not.
Key claims
SemVerBench found predictable version-constraint blind spots across npm, PEP 440 and Cargo—and a resolver solved the task.
Qualification: Supplying the governing rule recovered many errors; interval decomposition did not.
Evidence: source-2026-09-12-005
Sources
- arXiv preprint 2609.11180arXiv · primary research
Corrections
No corrections have been recorded for this story.