Ensemble Variability as a Signal of Confounding in Medical Imaging Models

Machine learning models for medical image analysis are vulnerable to hidden confounders, which can compromise generalization and clinical reliability. Existing detection strategies typically require explicit knowledge or labels of the confounder, which are often unavailable. In this work, we propose an ensemble-based framework to detect potential confounder-driven learning without explicitly defining the confounders, but only which samples might be affected. Our approach leverages the variability of model performance across ensembles to identify signatures of shortcut learning. Shortcut learning occurs when a model uses non-robust features or correlations rather than learning the true underlying task, and it is often observed when confounders are present. We generate controlled dataset variants with increasing confounding levels and analyze distributions of AUC (area under the ROC curve) scores across training, validation, and test splits, revealing converging performance and reduced variance as confounding intensifies. We validate our method on two clinically relevant tasks, diabetic retinopathy detection from retinal fundus images and tumor detection from brain MRI slices. Then, we further demonstrate its practical utility on another dataset and image modality with a stroke reperfusion prediction task with suspected hidden confounders. This work provides a practical, data-driven diagnostic tool to flag potential confounding and support the reliability assessment of machine learning models in medical imaging.