benchmarks evals
Test-Set Tuning Moved Scores More Than Rankings
Experiments found real performance inflation but often preserved which models ranked ahead.
Summary
Experiments found real performance inflation but often preserved which models ranked ahead.
Across MNIST-1D, CIFAR-10 and three GLUE tasks, tuning hyperparameters on the test set produced statistically significant optimism, yet the magnitude was often small beside other noise and model rankings were largely preserved. The authors argue for transparent reporting and a more measured diagnosis—not permission to erase the validation boundary.
Why it matters
Experiments found real performance inflation but often preserved which models ranked ahead.
Limits and context
- The authors argue for transparent reporting and a more measured diagnosis—not permission to erase the validation boundary.
Key claims
Experiments found real performance inflation but often preserved which models ranked ahead.
Qualification: The authors argue for transparent reporting and a more measured diagnosis—not permission to erase the validation boundary.
Evidence: source-2026-10-06-021
Sources
- arXiv preprint 2610.05902arXiv · primary research
Corrections
No corrections have been recorded for this story.