Classification Performance of a Deep Learning Algorithm for Common Skin Lesions in the Emergency Department: A Multicenter Preliminary Validation Study

Background/Objectives: Initial classification of skin lesions in the emergency department (ED) is difficult when dermatology consultations are limited. This study externally validated a deep learning (DL) algorithm for classifying three common ED skin lesions and explored whether algorithmic pre-classification was associated with changes in the classification performance of medical staff. Methods: A ResNet-50-based algorithm was developed using 6000 clinical images from 4815 patients. For external validation, 100 images from an independent institution were classified by the algorithm and nine medical staff members (three emergency medicine specialists, three residents, and three physician assistant [PA] nurses). Fifteen days later, the staff reclassified the same randomly reordered images with the algorithmic pre-classification provided. Results: In external validation, the algorithm showed an overall accuracy of 0.96 (95% confidence interval [CI], 0.90–0.99), lesion-specific balanced accuracies of 0.95–0.96, and a macro-average area under the receiver operating characteristic curve of 0.98 (95% CI, 0.95–1.00). Unassisted, the three PA nurses differed significantly from the algorithm for urticaria and herpes zoster (all p < 0.01, McNemar’s exact test). After receiving algorithmic assistance, all medical staff showed a balanced accuracy of 0.97 or higher for all three lesions. In the mixed-effects model, assistance was associated with higher odds of correct classification (odds ratio 25.9, 95% CI 11.7–57.2), and inter-rater agreement (Fleiss’ κ) increased from 0.70 to 0.97. Conclusions: The algorithm demonstrated high classification performance on independent external validation, and algorithmic assistance was associated with improved classification performance, with the largest absolute gains among medical staff with lower unassisted accuracy. As these findings are exploratory, they should be confirmed in randomized prospective studies.