Out-of-Distribution Generalization of Pretrained Multi-Label Chest X-Ray Classifiers: A Retrospective External Validation of Seven Models
Background/Objectives: Chest radiograph classifiers are usually benchmarked within their training distribution, whereas deployment exposes them to new populations, equipment, and labeling conventions. This study quantified out-of-distribution (OOD) performance of seven public pretrained classifiers. Methods: Six DenseNet-121 models and one ResNet-50 TorchXRayVision model were evaluated without fine-tuning on 18,424 Brazilian BRAX studies for seven common labels. The in-distribution (ID) performances of the three models trained on ChestX-ray14 were measured on held-out patients. Areas under the receiver operating characteristic curve (AUROC) were compared with a paired patient-level bootstrap; calibration, operating points, and subgroups were assessed. Results: OOD macro AUROC was 0.733–0.741 for three models trained with PadChest and 0.652–0.677 for four single-source models (all between-group differences significant). Relative to held-out ID performance, macro AUROC decreased by 6.0–11.4 percentage points. Expected calibration error of default outputs (0.028–0.392) largely reflected score normalization and fell to 0.001–0.003 after recalibration; sensitivity at default operating points was 0.035–0.797. Macro AUROC was lower for anteroposterior than posteroanterior radiographs (0.499–0.605 vs. 0.691–0.796), also within age groups; residual confounding remains possible. Conclusions: In this single Brazilian cohort, ID benchmarks overestimated external performance, and default outputs and operating points did not transfer. Local validation, recalibration, and subgroup-stratified reporting are needed before clinical use.
Authors
- Justas Trinkūnas (ORCID: https://orcid.org/0000-0002-6559-049X)
- Paulius Bundza (ORCID: https://orcid.org/0009-0009-7823-8055)
Institutions
- Vilnius Gediminas Technical University (LT)
Publication Details
- Journal
- Diagnostics
- Published
- 2026-11-18
- DOI
- https://doi.org/10.3390/diagnostics16223865
- Primary Topic
- COVID-19 diagnosis using AI
- Type
- article
- Field-Weighted Citation Impact
- 0.00