research
The Diagnostic Benchmark Knew Too Much About Its Answers
Strict cross-dataset isolation cut mean Recall@1 from 54.3 to 40.3 percent in a reconstructed rare-disease agentic framework.
Summary
Strict cross-dataset isolation cut mean Recall@1 from 54.3 to 40.3 percent in a reconstructed rare-disease agentic framework.
High benchmark scores for augmented language-model diagnosis can reflect evaluation structure rather than clinical reasoning. An audit covering more than 60,000 cases reconstructed a state-of-the-art agentic framework and separated candidate sets across datasets. Mean Recall@1 fell from 54.3 percent under conventional one-versus-rest evaluation to 40.3 percent under strict isolation. Supervised classifiers showed a second generalization gap, declining from 42.9 percent internally to 16.7 percent across external cohorts. The authors also found that retrospective treatment cues can reveal diagnoses and that time mismatches in knowledge bases move results. This does not establish clinical performance for every system; it shows why rare-disease evaluations need data isolation, temporal controls and safeguards against therapeutic clues.
Why it matters
Strict cross-dataset isolation cut mean Recall@1 from 54.3 to 40.3 percent in a reconstructed rare-disease agentic framework.
Limits and context
- This does not establish clinical performance for every system; it shows why rare-disease evaluations need data isolation, temporal controls and safeguards against therapeutic clues.
Key claims
Strict cross-dataset isolation cut mean Recall@1 from 54.3 to 40.3 percent in a reconstructed rare-disease agentic framework.
Qualification: This does not establish clinical performance for every system; it shows why rare-disease evaluations need data isolation, temporal controls and safeguards against therapeutic clues.
Evidence: source-2026-10-27-005
Sources
- arXiv preprint 2610.33592, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.