developer tools
The Repository Changed and the Skill Stayed Quiet
Every one of 105 selected release transitions invalidated part of the prior repository skill set.
Summary
Every one of 105 selected release transitions invalidated part of the prior repository skill set.
Across 57 repositories, six frontier agents reached only 29.9 to 69.7 percent avg@3 macro F1 when asked to update V1 skills from official V1-to-V2 patches. Missed files left stale instructions intact, while broader edits raised recall at the cost of precision. This final text-only report fills the front rail beneath the feature.
Why it matters
Every one of 105 selected release transitions invalidated part of the prior repository skill set.
Limits and context
- Across 57 repositories, six frontier agents reached only 29.9 to 69.7 percent avg@3 macro F1 when asked to update V1 skills from official V1-to-V2 patches.
- This final text-only report fills the front rail beneath the feature.
Key claims
Every one of 105 selected release transitions invalidated part of the prior repository skill set.
Qualification: Across 57 repositories, six frontier agents reached only 29.9 to 69.7 percent avg@3 macro F1 when asked to update V1 skills from official V1-to-V2 patches.
Evidence: source-2026-08-25-016
Sources
- arXiv preprint 2608.21964arXiv · primary research
Corrections
No corrections have been recorded for this story.