CoDE: A Patient Cohort-Based Study of Evidence Coverage and Support in Diagnostic Explanations Generated by Large Language Models

Large language models (LLMs) generate medical diagnostic explanations using selected patient findings, raising the question of whether the cited evidence adequately supports the proposed diagnosis. Cohort models identify informative findings, but their value for evaluating independently generated explanations requires separate testing. We introduce CoDE (Cohort-based Diagnostic Evidence evaluation), a controlled empirical study comparing importance-weighted evidence coverage with joint diagnostic support from cited findings. On synthetic DDXPlus records and clinical-record-derived TRACE data, we examine evidence selection, generated explanations and controlled evidence changes using quantity-matched controls and separate evaluation predictors. Cohort importance improved selection, and guided evidence edits in DDXPlus changed diagnostic support more than matched random operations. For correct TRACE proposals, local positive-SHAP coverage exceeded uniform group coverage by 0.2371 failure-detection average precision (simultaneous 95% CI, 0.1093–0.3650), while tree-based direct support performed better. A second generator reproduced weighting gains, with uncertain ordering against direct support. Sparse failures and mapping sensitivity limited natural-explanation rankings in DDXPlus. These findings support using importance to prioritize evidence checks and joint support to assess evidence combinations. Potential omissions warrant testing by adding uncited evidence and measuring support changes; low coverage alone does not establish a reasoning error.