Cross-Dataset Reliability of Chest Radiograph Models: A Provenance-Aware Evaluation of Performance and Thresholds
Background Public chest radiograph datasets attach different label meanings and sources to findings such as pneumonia and lung opacity. A shared label name does not ensure that models are being evaluated against the same target. Objective The objective of this study was to compare discrimination, calibration, high-confidence errors, and source-threshold flag volume across four public chest radiograph tasks. Methods Four dataset-defined tasks were evaluated: Kaggle/Kermany pneumonia, Radiological Society of North America (RSNA) lung opacity, CheXpert pneumonia, and CheXpert lung opacity. For each image, probabilities from five saved DenseNet-121 seeds per source task were averaged. Each resulting ensemble was evaluated on all four held-out target splits, producing 16 comparisons. Performance measures included area under the receiver operating characteristic curve (AUROC), area under precision-recall curve (AUPRC), balanced accuracy, Brier score, expected calibration error with 15 equal-width bins (ECE15), negative log-likelihood, high-confidence errors, and flags per 1000 images at source-validation Youden thresholds. Percentile intervals were estimated from 500 target-set bootstrap replicates, resampling patients where identifiers were available and images otherwise. Results Same-label pneumonia transfer produced ensemble AUROCs of 0.699 (95% CI, 0.664-0.732) from Kaggle/Kermany to CheXpert and 0.758 (0.718-0.797) in the reverse direction. Opacity transfer produced AUROCs of 0.824 (0.808-0.841) from RSNA to CheXpert and 0.835 (0.820-0.847) in the reverse direction. ECE15 ranged from 0.053 to 0.533. For Kaggle pneumonia to RSNA opacity, 28.4% (1096/3853) of all images were both incorrect and at least 0.90 confident. On CheXpert opacity, Kaggle- and RSNA-derived thresholds flagged 959.5 and 348.0 images per 1000, respectively. Conclusion Model ranking, probability error, confident mistakes, and threshold-based flag volume varied across dataset-defined tasks. These results support reporting complementary measures and checking label provenance when evaluating image models across datasets. They do not establish clinical effectiveness or validate a clinical workflow.
Authors
- Murtaza Hussain Shaikh
- Sadman Haque Sakib
- Irin Jahan Sorna
- Sk Rakib Ul Islam Rahat
- Md Anik Hosan
- Najmuddin .
- Sharmin Akter Poppy
Institutions
- Kyungsung University (KR)
- Dong-Eui University (KR)
Publication Details
- Journal
- Cureus
- Published
- 2026-11-17
- DOI
- https://doi.org/10.7759/cureus.119832
- Primary Topic
- COVID-19 diagnosis using AI
- Type
- article
- Field-Weighted Citation Impact
- 0.00