Cross-Dataset Reliability of Chest Radiograph Models: A Provenance-Aware Evaluation of Performance and Thresholds

Background Public chest radiograph datasets attach different label meanings and sources to findings such as pneumonia and lung opacity. A shared label name does not ensure that models are being evaluated against the same target. Objective The objective of this study was to compare discrimination, calibration, high-confidence errors, and source-threshold flag volume across four public chest radiograph tasks. Methods Four dataset-defined tasks were evaluated: Kaggle/Kermany pneumonia, Radiological Society of North America (RSNA) lung opacity, CheXpert pneumonia, and CheXpert lung opacity. For each image, probabilities from five saved DenseNet-121 seeds per source task were averaged. Each resulting ensemble was evaluated on all four held-out target splits, producing 16 comparisons. Performance measures included area under the receiver operating characteristic curve (AUROC), area under precision-recall curve (AUPRC), balanced accuracy, Brier score, expected calibration error with 15 equal-width bins (ECE15), negative log-likelihood, high-confidence errors, and flags per 1000 images at source-validation Youden thresholds. Percentile intervals were estimated from 500 target-set bootstrap replicates, resampling patients where identifiers were available and images otherwise. Results Same-label pneumonia transfer produced ensemble AUROCs of 0.699 (95% CI, 0.664-0.732) from Kaggle/Kermany to CheXpert and 0.758 (0.718-0.797) in the reverse direction. Opacity transfer produced AUROCs of 0.824 (0.808-0.841) from RSNA to CheXpert and 0.835 (0.820-0.847) in the reverse direction. ECE15 ranged from 0.053 to 0.533. For Kaggle pneumonia to RSNA opacity, 28.4% (1096/3853) of all images were both incorrect and at least 0.90 confident. On CheXpert opacity, Kaggle- and RSNA-derived thresholds flagged 959.5 and 348.0 images per 1000, respectively. Conclusion Model ranking, probability error, confident mistakes, and threshold-based flag volume varied across dataset-defined tasks. These results support reporting complementary measures and checking label provenance when evaluating image models across datasets. They do not establish clinical effectiveness or validate a clinical workflow.

Authors

Institutions

Publication Details

Journal
Cureus
Published
2026-11-17
DOI
https://doi.org/10.7759/cureus.119832
Primary Topic
COVID-19 diagnosis using AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Cross-Dataset Reliability of Chest Radiograph Models: A Provenance-Aware Evaluation of Performance and Thresholds

Murtaza Hussain Shaikh, Sadman Haque Sakib, Irin Jahan Sorna, Sk Rakib Ul Islam Rahat et al.
2 links
Cureus
COVID-19 diagnosis using AI
article

Cross-Dataset Reliability of Chest Radiograph Models: A Provenance-Aware Evaluation of Performance and Thresholds

Murtaza Hussain Shaikh, Sadman Haque Sakib, Irin Jahan Sorna, Sk Rakib Ul Islam Rahat, Md Anik Hosan, Najmuddin ., Sharmin Akter Poppy
article en

Abstract

Background Public chest radiograph datasets attach different label meanings and sources to findings such as pneumonia and lung opacity. A shared label name does not ensure that models are being evaluated against the same target. Objective The objective of this study was to compare discrimination, calibration, high-confidence errors, and source-threshold flag volume across four public chest radiograph tasks. Methods Four dataset-defined tasks were evaluated: Kaggle/Kermany pneumonia, Radiological Society of North America (RSNA) lung opacity, CheXpert pneumonia, and CheXpert lung opacity. For each image, probabilities from five saved DenseNet-121 seeds per source task were averaged. Each resulting ensemble was evaluated on all four held-out target splits, producing 16 comparisons. Performance measures included area under the receiver operating characteristic curve (AUROC), area under precision-recall curve (AUPRC), balanced accuracy, Brier score, expected calibration error with 15 equal-width bins (ECE15), negative log-likelihood, high-confidence errors, and flags per 1000 images at source-validation Youden thresholds. Percentile intervals were estimated from 500 target-set bootstrap replicates, resampling patients where identifiers were available and images otherwise. Results Same-label pneumonia transfer produced ensemble AUROCs of 0.699 (95% CI, 0.664-0.732) from Kaggle/Kermany to CheXpert and 0.758 (0.718-0.797) in the reverse direction. Opacity transfer produced AUROCs of 0.824 (0.808-0.841) from RSNA to CheXpert and 0.835 (0.820-0.847) in the reverse direction. ECE15 ranged from 0.053 to 0.533. For Kaggle pneumonia to RSNA opacity, 28.4% (1096/3853) of all images were both incorrect and at least 0.90 confident. On CheXpert opacity, Kaggle- and RSNA-derived thresholds flagged 959.5 and 348.0 images per 1000, respectively. Conclusion Model ranking, probability error, confident mistakes, and threshold-based flag volume varied across dataset-defined tasks. These results support reporting complementary measures and checking label provenance when evaluating image models across datasets. They do not establish clinical effectiveness or validate a clinical workflow.

Cureus
Kyungsung University (KR), Dong-Eui University (KR)
Good health and well-being
COVID-19 diagnosis using AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon