benchmarks evals
A Tenfold Time-Horizon Jump Was Not Always the Same Leap
A statistical reanalysis finds that human task time maps unevenly onto AI difficulty, especially between two and thirty minutes.
Summary
A statistical reanalysis finds that human task time maps unevenly onto AI difficulty, especially between two and thirty minutes.
METR’s time horizon expresses capability as the human completion time of tasks an AI solves with 50% probability. Reanalyzing 228 tasks and 26 systems with splines and item-response theory, the authors find that the mapping from human time to AI difficulty is nearly flat from roughly two to thirty minutes and closer to linear elsewhere. That makes a jump from three to thirty minutes statistically easier than one from thirty minutes to five hours, despite the same multiplier. Cross-validated scoring favored the new estimates. The paper does not invalidate time horizons; it argues that point estimates should travel with diagnostic plots and construct-validity checks.
Why it matters
A statistical reanalysis finds that human task time maps unevenly onto AI difficulty, especially between two and thirty minutes.
Limits and context
- The paper does not invalidate time horizons; it argues that point estimates should travel with diagnostic plots and construct-validity checks.
Key claims
A statistical reanalysis finds that human task time maps unevenly onto AI difficulty, especially between two and thirty minutes.
Qualification: The paper does not invalidate time horizons; it argues that point estimates should travel with diagnostic plots and construct-validity checks.
Evidence: source-2026-10-10-006
Sources
- arXiv preprint 2610.12466, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.