benchmarks evals
The Proof Judge Cuts False Positives From 23 to 6 Percent
A pipeline for natural-language proofs, computer-algebra code and Lean formalizations found distinct solutions—and a proof of one published open conjecture—in Ramanujan Challenge submissions.
Summary
A pipeline for natural-language proofs, computer-algebra code and Lean formalizations found distinct solutions—and a proof of one published open conjecture—in Ramanujan Challenge submissions.
Research-level mathematics creates two problems for AI: proposing a proof and judging it. The Ramanujan Challenge analysis reports correct submissions distinct from official solutions and a proof of one published open conjecture, then evaluates a structured judging pipeline across prose, code and formal proofs. Compared with one-shot model judgment, the pipeline reduced false positives on natural-language proofs from 23 percent to 6 percent and raised supporting-code judgment accuracy from 46 percent to 58 percent. Those figures describe the challenge submissions and evaluation set, not a general guarantee that automated proof review is reliable.
Why it matters
A pipeline for natural-language proofs, computer-algebra code and Lean formalizations found distinct solutions—and a proof of one published open conjecture—in Ramanujan Challenge submissions.
Limits and context
- Those figures describe the challenge submissions and evaluation set, not a general guarantee that automated proof review is reliable.
Key claims
A pipeline for natural-language proofs, computer-algebra code and Lean formalizations found distinct solutions—and a proof of one published open conjecture—in Ramanujan Challenge submissions.
Qualification: Those figures describe the challenge submissions and evaluation set, not a general guarantee that automated proof review is reliable.
Evidence: source-2026-10-31-003
Sources
- arXiv preprint 2610.40551, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.