Auxiliary diagnosis of hyperpigmented skin diseases using multimodal deep learning

To the Editor: Hyperpigmented skin diseases (HSDs) are characterized primarily by excessive melanin production or an increase in melanocyte density, commonly manifesting as melasma, freckles, and café-au-lait spots.[1] These conditions are often multifaceted, with a prolonged course and a high tendency for recurrence. HSDs frequently manifest in exposed areas such as the face, affecting both appearance and quality of life, and may lead to psychological issues such as insomnia, anxiety, and depression. Moreover, the clinical features of some HSDs can resemble those of malignant skin diseases such as melanoma, thereby increasing diagnostic challenges and the psychological burdens of patients.[2] The integration of multimodal information for disease diagnosis is a common practice in clinical settings.[3] Dermatologists identify the features of hyperpigmented skin disorders primarily through visual examination while considering patient demographic information (such as age and sex), as well as symptoms and physical examination findings, to perform a comprehensive analysis. However, owing to a lack of clinical experience, primary care physicians often struggle to accurately differentiate similar skin lesions, especially when multiple lesions are present in the same area, which increases the risk of misdiagnosis. Therefore, the development of convenient and efficient auxiliary diagnostic tools is highly important. This study proposes a multimodal diagnostic (MDD) model that integrates clinical skin images, narrative text, and structured data, aiming to aid primary dermatologists by improving diagnostic accuracy and reducing misdiagnoses. We retrospectively collected clinical data from 1536 patients at the Dermatology Laser Medicine Center of the Air Force Medical Center of the Chinese People’s Liberation Army. The study protocol was approved by the Ethics Committee of the Air Force Medical Center of the Chinese People’s Liberation Army (No. 2022-227-YJ01). Informed consent was obtained from all participants. The dataset includes 6142 clinical skin images and 1536 clinical history records. The patients were treated at the center between 2013 and 2023 and were diagnosed with conditions such as nevus fuscoceruleus (NF), freckles (FC), melasma (MLA), seborrheic keratosis (SK), pigmented nevus (PN), lentigo (LE), pigmented hairy epidermal nevus (PHEN), melanosis (MLS), nevus of Ota (NO), and café-au-lait spot (CS). Data preprocessing included image de-identification, image segmentation, and text data cleaning, among other procedures. The final dataset was split into training and validation sets at a 4:1 ratio. The overall architecture of the MDD model comprises three key components: a multimodal feature extraction module, a feature fusion module, and a classification module. Details of the data processing, model development, and validation procedures are outlined in the Supplementary Methods, Supplementary Figures 1 and 2, https://links.lww.com/CM9/C457. We included 1536 patients (mean age, 23.4 ± 14.5 years; range, 0–80.0 years) in the study, with a total of 6142 skin images and 1536 medical history records. The training cohort comprised 1224 patients (mean age, 23.6 ± 14.7 years; range, 0–78.0 years) and 4879 lesion images (mean diameter, 14.95 ± 24.04 cm²), and the validation cohort consisted of 312 patients (mean age, 23.6 ± 14.7 years; range, 0–72.0 years) and 1263 lesion images (mean diameter, 16.58 ± 23.61 cm²). PN lesions accounted for 32.49% of the patients included in the study, NO accounted for 18.10%, FC accounted for 10.61%, MLA accounted for 4.75%, and CSs accounted for 15.10%. Other conditions included NF, SK, LE, PHEN, and MLS, which accounted for 7.37%, 2.73%, 1.89%, 1.32%, and 0.52%, respectively. A detailed overview of the baseline demographic and disease characteristics of the study population can be found in Supplementary Tables 1 and 2, https://links.lww.com/CM9/C457. For the diagnostic performance evaluation, we used several quantitative metrics—the area under the receiver operating characteristic (ROC) curve (AUC), accuracy (Acc), sensitivity (Sen), specificity (Spe), positive predictive value (PPV), and negative predictive value (NPV)—to assess each model. Compared with the other models, the trimodal MDD model had a marked advantage in terms of Acc, Sen, Spe, PPV, and NPV [Table 1]. Furthermore, among the four models evaluated with ROC curves, the unimodal model, which relies solely on clinical skin images, demonstrated relatively low diagnostic precision (AUC = 0.9554). In contrast, Bimodal Model 1 (AUC = 0.9762) and Bimodal Model 2 (AUC = 0.9802) both exhibited improved diagnostic performance. The MDD model achieved the greatest diagnostic efficacy (AUC = 0.9848), confirming that integrating multiple data sources significantly increases diagnostic accuracy [Supplementary Figure 3, https://links.lww.com/CM9/C457]. Table 1 - Acc, Sen, Spe, PPV, NPV, and AUC for the 10 classifications with the MDD model using the validation set. Category Acc (95% CI) Sen (95% CI) Spe (95% CI) PPV (95% CI) NPV (95% CI) AUC (95% CI) PN 0.9715 (0.9608, 0.9800) 0.9450 (0.9134, 0.9676) 0.9801 (0.9691, 0.9880) 0.9389 (0.9062, 0.9628) 0.9821 (0.9716, 0.9896) 0.9912 (0.9839, 0.9986) NO 0.9470 (0.9331, 0.9587) 0.8561 (0.8099, 0.8947) 0.9734 (0.9613, 0.9826) 0.9037 (0.8621, 0.9361) 0.9587 (0.9444, 0.9702) 0.9751 (0.9623, 0.9879) FC 0.9786 (0.9690, 0.9859) 0.9400 (0.8892, 0.9722) 0.9838 (0.9746, 0.9904) 0.8868 (0.8270, 0.9315) 0.9918 (0.9846, 0.9963) 0.9926 (0.9829, 1.0000) MLA 0.9881 (0.9805, 0.9933) 0.9718 (0.9353, 0.9908) 0.9908 (0.9831, 0.9956) 0.9451 (0.9013, 0.9733) 0.9954 (0.9892, 0.9985) 0.9974 (0.9922, 1.0000) CS 0.9549 (0.9419, 0.9656) 0.8311 (0.7608, 0.8876) 0.9713 (0.9597, 0.9803) 0.7935 (0.7212, 0.8543) 0.9774 (0.9669, 0.9853) 0.9774 (0.9606, 0.9943) NF 0.9865 (0.9785, 0.9921) 0.8000 (0.6823, 0.8890) 0.9967 (0.9915, 0.9991) 0.9286 (0.8271, 0.9802) 0.9892 (0.9817, 0.9943) 0.9903 (0.9734, 1.0000) SK 0.9857 (0.9776, 0.9915) 0.8600 (0.7326, 0.9418) 0.9909 (0.9838, 0.9955) 0.7963 (0.6647, 0.8937) 0.9942 (0.9881, 0.9977) 0.9873 (0.9654, 1.0000) LE 0.9850 (0.9766, 0.9909) 0.5000 (0.3243, 0.6757) 0.9984 (0.9941, 0.9998) 0.8947 (0.6686, 0.9870) 0.9863 (0.9782, 0.9920) 0.9583 (0.9112, 1.0000) PHEN 0.9881 (0.9805, 0.9933) 0.8667 (0.5954, 0.9834) 0.9896 (0.9823, 0.9944) 0.5000 (0.2993, 0.7007) 0.9984 (0.9942, 0.9998) 0.9911 (0.9576, 1.0000) MLS 0.9913 (0.9845, 0.9956) 0.8333 (0.6528, 0.9436) 0.9951 (0.9894, 0.9982) 0.8065 (0.6253, 0.9255) 0.9959 (0.9906, 0.9987) 0.9848 (0.9538, 1.0000) Acc: Accuracy; AUC: Area under the ROC curve; CS: Café-au-lait spot; FC: Freckles; LE: Lentigo; MDD: Multimodal Diagnostic; MLA: Melasma; MLS: Melanosis; NF: Nevus fuscoceruleus; NO: Nevus of Ota; NPV: Negative predictive value; PHEN: Pigmented hairy epidermal nevus; PN: pigmented nevus; PPV: Positive predictive value; Sen: Sensitivity; SK: Seborrheic keratosis; Spe: Specificity. To assess the classification performance of the MDD model, we used confusion matrices, which revealed that the MDD model results in the fewest classification errors and that its predictions align closely with the actual outcomes [Supplementary Figure 4, https://links.lww.com/CM9/C457]. In addition, we used heatmaps to highlight key regions of interest, thereby providing interpretability for the MDD model’s predictions. These heatmaps emphasize the lesion areas on which the model focuses when diagnosing pigmented skin diseases, which closely match the areas that clinicians consider most relevant. This alignment suggests that the MDD model effectively captures the critical features of the disease and is consistent with clinical diagnoses [Supplementary Figure 5, https://links.lww.com/CM9/C457]. Although this study produced notable results, several limitations remain. First, the uneven sample distribution may affect the model’s generalizability. Second, the model was developed primarily based on data from a Chinese population, and its applicability to other populations requires further validation. Third, the lack of multicenter external validation constrains the broader applicability of the study. Future work will expand data collection and foster multicenter collaborations to encompass a wider range of populations, thereby improving the model’s robustness and generalizability.[4] In conclusion, by incorporating clinical skin images, structured data, and narrative text, the MDD model has markedly improved diagnostic accuracy for hyperpigmented skin disorders, allowing its use as a valuable diagnostic aid in primary care. This study highlights the potential value of multimodal artificial intelligence in dermatology. Funding This work was supported by a grant from the Air Force Logistics Department (No. BKJW221J009). Acknowledgments We extend our sincere gratitude to the entire staff of the Laser Center, Department of Dermatology at the Air Force Med­ical Center of the Chinese People’s Liberation Army, for their invaluable assistance throughout this work. Conflicts of interest None.