Multi-criterion uncertainty estimation improves skin cancer distribution shift detection and malignancy prediction

Abstract Accessible imaging technologies and stronger machine learning (ML) architectures have spurred a race to develop models to complete dermatological tasks like automated skin cancer diagnosis. However, models with high performance on benchmark datasets deteriorate when challenged with data from disparate clinical sources. Generalization gaps stem from high variability in skin lesion images from capture angle, imaging technology, and patient phenotype, among other factors, impeding the safe application of ML models. We apply a novel multi-criterion uncertainty-estimation approach to detect out-of-distribution skin lesion images from five publicly available datasets across seven countries. Using our method, Supervised Autoencoders for Generalization Estimates (SAGE), we quantify likeness of images from patients in Argentina, Brazil, Austria, North Macedonia, Turkey, Australia, and the United States to the popular HAM10000 benchmarking dataset and identify problematic image artifacts affecting the reliability of predictions in a pre-clinical setting. We show how filtering images based on SAGE score can improve the performance of a separate malignancy prediction model. We benchmark SAGE across varying severities of distribution shift (e.g., image modality and new diagnostic classes), providing users with a powerful tool for interrogating differences between their data and the training distribution of an ML model before clinical implementation.