A Multimodal Approach for Skin Lesion Diagnosis

Inspection by dermatologists remains the standard for skin cancer diagnosis, but it can be subjective and prone to error. We explore the potential of machine learning for dermoscopic image classification using 10,000 pigmented skin lesions. Although convolutional neural networks are powerful tools for skin lesion classification by themselves, by incorporating patient-specific textual data, we aim to improve the accuracy and accessibility of diagnosis. We propose multiple multimodal models: transfer learning and multimodal transformers. These multimodal approaches aim to utilise both dermoscopic images and corresponding patient-specific data, such as lesion location and patient demographics. Our multimodal model with the best overall evaluation is the multimodal transformer model which combines a vision transformer and a text transformer. It achieved an accuracy of 93.21% and an AUROC of 0.9860.