research
Six Agents Built Climate Models That Beat the Published Baseline
A hidden-grader exam tested whether open-ended agent research reproduced ENSO statistics, inferred unseen variables and forecast held-out years.
Summary
A hidden-grader exam tested whether open-ended agent research reproduced ENSO statistics, inferred unseen variables and forecast held-out years.
SciExam for ENSO gives agents six hours to process real observations, write diagnostics that are then frozen and build low-order stochastic climate models using only those diagnostics as feedback. Hidden graders evaluate statistics, unobserved-variable recovery and held-out forecasts, and score a published model under the same procedure. Six of twelve agent systems scored higher than that baseline, mainly through reconstruction and forecasting. Simplified versions of the stronger models aligned with the two competing explanations of ENSO's warm-cold asymmetry even though the prompt did not mention the debate. The benchmark does not establish new climate science by itself, but it offers an unusually demanding test: research output is judged by unseen physical behavior rather than a known answer or another model's review.
Why it matters
A hidden-grader exam tested whether open-ended agent research reproduced ENSO statistics, inferred unseen variables and forecast held-out years.
Limits and context
- SciExam for ENSO gives agents six hours to process real observations, write diagnostics that are then frozen and build low-order stochastic climate models using only those diagnostics as feedback.
- Simplified versions of the stronger models aligned with the two competing explanations of ENSO's warm-cold asymmetry even though the prompt did not mention the debate.
- The benchmark does not establish new climate science by itself, but it offers an unusually demanding test: research output is judged by unseen physical behavior rather than a known answer or another model's review.
Key claims
A hidden-grader exam tested whether open-ended agent research reproduced ENSO statistics, inferred unseen variables and forecast held-out years.
Qualification: SciExam for ENSO gives agents six hours to process real observations, write diagnostics that are then frozen and build low-order stochastic climate models using only those diagnostics as feedback.
Evidence: source-2026-10-08-011
Sources
- arXiv preprint 2610.10513arXiv · primary research
Corrections
No corrections have been recorded for this story.