benchmarks evals
The Typed Judge Followed the Option Name
Schema-valid outputs still reversed when semantically loaded option labels changed.
Published Updated Story ID: mp-2026-09-23-015
Summary
Schema-valid outputs still reversed when semantically loaded option labels changed.
Across 1,200 workflow decisions, renaming identical rubrics from neutral options to no/yes changed 70.4 more answers per hundred in one tested setup while type errors stayed at zero. Random strings returned behavior to the neutral regime, implicating semantic option names rather than schema validity.
Why it matters
Schema-valid outputs still reversed when semantically loaded option labels changed.
Limits and context
No additional limitation was separately recorded.
Key claims
Schema-valid outputs still reversed when semantically loaded option labels changed.
Evidence: source-2026-09-23-017
Sources
- arXiv preprint 2609.26758arXiv · primary research
Corrections
No corrections have been recorded for this story.