benchmarks evals
Change the Rating Scale, Change One in Five Labels
Across five models and three social-science tasks, one scale-design change moved normalized scores by about 0.10 on average.
Summary
Across five models and three social-science tasks, one scale-design change moved normalized scores by about 0.10 on average.
Roughly 20 percent of texts changed category across alternative formats, ranges or signs, with the largest sensitivity on toxicity. Rubric prompting did not remove the effect. The finding argues for robustness checks; it does not invalidate every model-based score.
Why it matters
Across five models and three social-science tasks, one scale-design change moved normalized scores by about 0.10 on average.
Limits and context
- Rubric prompting did not remove the effect.
- The finding argues for robustness checks; it does not invalidate every model-based score.
Key claims
Across five models and three social-science tasks, one scale-design change moved normalized scores by about 0.10 on average.
Qualification: Rubric prompting did not remove the effect.
Evidence: source-2026-10-17-021
Sources
- arXiv preprint 2610.22187, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.