benchmarks evals
Research Agents Propose Useful Experiments. Ranking Them Is Harder.
Of 415 proposed minimum viable experiments, 194 were judged sound, valuable and new, but only 73 also moved beyond the seed paper and the obvious.
Summary
Of 415 proposed minimum viable experiments, 194 were judged sound, valuable and new, but only 73 also moved beyond the seed paper and the obvious.
RTaste asks whether research agents can choose which experiment deserves to run before seeing the outcome. Six frontier models proposed, forecast and ranked 415 minimum viable experiments based on seven recent NLP papers; a separate executor attempted every proposal, and model auditors judged the results. Auditors labeled 194 sound, valuable and new, but only 73 also beyond the seed paper and non-obvious. First choices beat random selection yet did not outperform simply choosing the first proposed experiment, and teams cost more without a clear overall quality gain. Because model auditors supplied the judgments, the harness is evidence about this setup rather than a definitive measure of scientific taste.
Why it matters
Of 415 proposed minimum viable experiments, 194 were judged sound, valuable and new, but only 73 also moved beyond the seed paper and the obvious.
Limits and context
- Auditors labeled 194 sound, valuable and new, but only 73 also beyond the seed paper and non-obvious.
- First choices beat random selection yet did not outperform simply choosing the first proposed experiment, and teams cost more without a clear overall quality gain.
Key claims
Of 415 proposed minimum viable experiments, 194 were judged sound, valuable and new, but only 73 also moved beyond the seed paper and the obvious.
Qualification: Auditors labeled 194 sound, valuable and new, but only 73 also beyond the seed paper and non-obvious.
Evidence: source-2026-11-01-014
Sources
- arXiv preprint 2610.40488, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.