research
The Citation Exists. It Still Supports the Wrong Claim.
Biomedical agents almost always produced real PubMed identifiers in closed-book mode, but almost none were relevant to the claim.
Summary
Biomedical agents almost always produced real PubMed identifiers in closed-book mode, but almost none were relevant to the claim.
A controlled SciFact study compared closed-book answers, single-shot PubMed retrieval, a query-writing agent and access to gold abstracts. Of 335 closed-book citations, 327 PubMed identifiers existed, yet an independent judge found only three relevant to the claim. Letting the agent write queries tripled gold-paper recall and reached 62 percent verdict accuracy; gold abstracts lifted the same models to 79 to 86 percent, pointing to retrieval as the main bottleneck. The realistic-harness accuracy differences were not statistically significant at the sample size.
Why it matters
Biomedical agents almost always produced real PubMed identifiers in closed-book mode, but almost none were relevant to the claim.
Limits and context
- Of 335 closed-book citations, 327 PubMed identifiers existed, yet an independent judge found only three relevant to the claim.
- The realistic-harness accuracy differences were not statistically significant at the sample size.
Key claims
Biomedical agents almost always produced real PubMed identifiers in closed-book mode, but almost none were relevant to the claim.
Qualification: Of 335 closed-book citations, 327 PubMed identifiers existed, yet an independent judge found only three relevant to the claim.
Evidence: source-2026-10-22-015
Sources
- arXiv preprint 2610.29812, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.