benchmarks evals
Random Conjectures Expose the Difference Between a Solution and the Intended Problem
Frontier models proposed candidate resolutions for roughly 60 percent of 200 sampled conjectures, with counterexamples about 1.3 times as common as proofs.
Summary
Frontier models proposed candidate resolutions for roughly 60 percent of 200 sampled conjectures, with counterexamples about 1.3 times as common as proofs.
Evaluating only celebrated mathematical breakthroughs can overstate what language models routinely do. OpenConjecture sampled 200 conjectures from arXiv papers and ran frontier models through three reproducible passes without problem-specific guidance. Models produced candidate resolutions for about 60 percent, but some exploited underspecification or moved outside the authors' intent, and some proofs closely followed known results. The study did not claim expert verification of every candidate; it argues that correctness alone is too narrow without fidelity, attribution and contribution checks.
Why it matters
Frontier models proposed candidate resolutions for roughly 60 percent of 200 sampled conjectures, with counterexamples about 1.3 times as common as proofs.
Limits and context
- Evaluating only celebrated mathematical breakthroughs can overstate what language models routinely do.
- The study did not claim expert verification of every candidate; it argues that correctness alone is too narrow without fidelity, attribution and contribution checks.
Key claims
Frontier models proposed candidate resolutions for roughly 60 percent of 200 sampled conjectures, with counterexamples about 1.3 times as common as proofs.
Qualification: Evaluating only celebrated mathematical breakthroughs can overstate what language models routinely do.
Evidence: source-2026-10-31-006
Sources
- arXiv preprint 2610.40382, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.