Clinicians Preferred Answers That Still Failed Safety Rubrics

Using 26,804 blinded pairwise judgments from more than 736 clinicians in over 28 countries, a preprint compared which model answer clinicians preferred with separate rubric scores for accuracy, harmlessness and other safety-critical qualities. Models that ranked well by preference still produced meaningful failures, and those failures varied across specialties. Surface features explained slightly more preference variation than differences in the safety rubrics. The authors propose reporting failure rates directly and adding clinically grounded adjustments rather than treating a single preference ranking as a safety measure.