Fine-tuning a large multimodal neural network with clinical and dermoscopic images for accurate detection and histopathological subtype prediction of basal cell carcinoma in long-tailed distributions

To the Editor: Basal cell carcinoma (BCC) is the most prevalent skin malignancy, with an increasing incidence and economic burden worldwide.[1] Various histopathological subtypes of BCC have been well described, and subtype confirmation is essential for BCC classification according to the risk of recurrence.[2] Early diagnosis and intervention are important, especially considering that the incidence of aggressive subtypes of BCC is increasing faster than that of indolent subtypes.[3] Initial screening for BCC relies mainly on clinical observation and dermoscopy, which can further assist in distinguishing between different histopathological subtypes. Recently, artificial intelligence (AI), especially convolutional neural networks (CNNs), has been successfully applied to classify BCC.[4] However, the diagnostic performance of models based on single-modal light-skinned images is limited, with a risk of impaired accuracy when these models are applied to patients with darker skin. Therefore, we aimed to fine-tune a multimodal neural network using dermoscopic and clinical images of Chinese patients to assist in the differentiation between BCC and common benign tumors, including melanocytic nevus (MN), seborrheic keratosis (SK), and dermatofibroma (DF), and further predict the subtypes of BCC. This study was approved by the Medical Ethics Committee of Peking Union Medical College Hospital (No. I-23PJ360). All the included patients or their parents (for the minors) signed written informed consent. Clinical and dermoscopic images of patients diagnosed with BCC, MN, SK, and DF that were captured from June 2018 to January 2024 were collected from the skin imaging database of our department, screened, and annotated by two experienced dermatologists independently. All the patients with BCC were further divided according to their histopathological subtypes. Two datasets were formed, with 2675 images (four classes, BCC vs. benign) and 594 images (six classes, BCC subtype distinction) [Supplementary Table 1, https://links.lww.com/CM9/C334]. The second dataset exhibited a pronounced long-tail distribution [Supplementary Figure 1, https://links.lww.com/CM9/C334] and addressing this issue is also a key focus of our proposed method. Our approach involved three main components. First, an information complementary block (ICB) improved classification by integrating unique features for better extraction efficiency [Supplementary Figure 2, https://links.lww.com/CM9/C334]. Second, we fine-tuned a large contrastive language-image pretraining (CLIP) model on our dataset, freezing most parameters and adjusting a smaller subset to enhance task-specific performance. Third, the logit-adjusted loss (LA loss) was used to address long-tail dataset imbalance by adjusting the Softmax output probabilities based on category distribution. We used ResNet as the feature extractor, which provided comprehensive lesion characterization. Features from shallow to deep layers were fused in the ICB module, with convolutional layers and a linear classification head for multiclass output. Experimental comparisons revealed that the performance was the best when features from Stages 2 and 4 were used. To address the long-tail distribution, we fine-tuned a CLIP-based model with an image encoder and a text encoder. The image encoder captured feature information, whereas the text encoder extracted categorical information. Using contrastive loss, we aligned image and category features. We selectively updated the image encoder parameters and introduced LA loss to increase the model’s focus on underrepresented categories, improving performance on tail categories [Supplementary Figure 3, https://links.lww.com/CM9/C334]. The performance of the proposed model is shown in detail in Supplementary Figure 4 and Supplementary Table 2, https://links.lww.com/CM9/C334. In the first task, the model exhibited near-perfect sensitivity and specificity across the four categories. In the second task, the model demonstrated better performance in discerning nodular (NOD), superficial (SUP), and mixed (MIX) BCC, with both sensitivity and specificity higher than 0.90, whereas its performance in distinguishing micronodular (MIC) and infiltrative (INF) subtypes and basosquamous carcinoma (BSQ) was weaker. The receiver operating characteristic (ROC) curves and the area under the curve (AUC) results are shown in Figure 1A. Confusion matrices are shown in Figure 1B. We compared our multimodal approach with other single-modal models [Supplementary Table 3, https://links.lww.com/CM9/C334]. The results showed that our multimodal model outperformed the single-modal models based on either clinical or dermoscopic images alone, highlighting the robustness and superior reliability of multimodal integration and leveraging the strengths of each modality. As the saliency map in Figure 1C shows, our model employs CNNs combined with gradient-weighted class activation mapping (Grad-CAM) to visualize the key regions associated with disease classification. The results demonstrate that the model has learned the correct lesion features. Figure 1D shows t-distributed stochastic neighbor embedding (t-SNE) visualizations of the internal features from our framework, revealing clear clusters of skin images by clinical category and demonstrating the model’s effectiveness in distinguishing different skin tumors and BCC subtypes.Figure 1: Performance and visualization of our proposed model. (A) ROC curves and the AUC results. Curves on the right demonstrate zoomed-in views between the abscissa 0–0.2. (B) Confusion matrices for the four-class classification task (left) and the six-class classification task (right). (C) Saliency map. (D) The t-SNE visualization for the four-class classification task (left) and the six-class classification task (right). AUC: Area under the curve; BCC: Basal cell carcinoma; BSQ: Basosquamous carcinoma; DF: Dermatofibroma; INF: Infiltrative BCC; MIC: Micronodular BCC; MIX: Mixed forms of BCC subtypes; MN: Melanocytic nevus; t-SNE: t-distributed stochastic neighbor embedding; NOD: Nodular BCC; ROC: Receiver operating characteristic; SK: Seborrheic keratosis; SUP: Superficial BCC.Given that the global population is aging persistently and that actinic damage accumulates, an accurate tool that helps with the rapid and convenient screening of suspicious BCC lesions is greatly needed, especially in rural areas where the insufficiency of dermatologists is critical. Multimodal fusion models aim to integrate data from various modalities to provide a more comprehensive understanding of complex scenarios. However, challenges such as long-tail distributions leading to biased performance on underrepresented classes and difficulties in aligning and fusing different modalities effectively exist. In this study, we propose a multimodal architecture on the basis of a CLIP-based model with LA loss, which utilizes clinical and dermoscopic images integrated with ICBs. The approach achieved rather high accuracy in the detection of BCC. In addition, innovatively, our model was able to predict different histopathological subtypes of BCC cases with an overall accuracy of 0.899, providing an important reference to clinicians for precise, individualized evaluation. Our model attempts to classify BCC cases into different histopathological subtypes. The BCC subtype of the patient, in combination with clinical information, is essential for predicting the risk of recurrence, staging, and making management plans.[5] However, several limitations exist. First, we could not include all rare cases, such as morpheaform BCC and fibroepithelial BCC. Second, we did not include other benign or malignant skin tumors, such as hemangioma and melanoma. Third, we could not validate our model with interactions with dermatologists or with external data. The inclusion of more types of skin tumors, external validation with images captured in various settings, and evaluation in real-world clinical settings in subsequent studies will be essential for further assessing the application value of our model. Funding This work was supported by grants from CAMS Innovation Fund for Medical Sciences (CIFMS) (No. 2022-I2M-C&T-A-007), the National Natural Science Foundation of China (No. 92354307), and the Fundamental Research Funds for the Central Universities (No. 2023RC09). Conflicts of interest None.