Enhancing Melanoma Detection through Multi-Modal Learning: Integration of Dermoscopic Images and Clinical Data

Melanoma is the deadliest form of skin cancer, and early detection is crucial for improving patient outcomes and survival rates. Traditional detection methods rely on visual inspections or single-modal Machine Learning (ML) techniques, which often face limitations in accuracy and generalizability. To address these challenges, this paper proposes a multi-modal learning framework that integrates dermoscopic images and clinical data for improved melanoma detection. The framework combines the visual patterns extracted from dermoscopic images using EfficientNet, a pretrained Convolutional Neural Network (CNN), with contextual insights of clinical data processed through a Multi-Layer Perceptron (MLP). An attention-based integration layer then merges the complementary features, prioritizing relevant information to enhance the performance of the model. Finally, the output layer supports multi-task learning by classifying lesions as benign or malignant and performing auxiliary tasks like risk estimation and lesion segmentation. The framework is trained and evaluated on a comprehensive dataset of dermoscopic images and clinical data, achieving significant improvements in key metrics such as accuracy (88%), precision ($66 \%$), recall ($80 \%$), and F -score ($72 \%$). The results demonstrate the ability of our model to minimize false negatives and false positives while maintaining robust detection performance. Compared to existing methods, our proposed model achieves superior performance exploiting the strengths of both data modalities. The findings highlight the potential of multi-modal learning for real-world clinical applications, providing a scalable and interpretable solution for early melanoma detection. This research contributes to advancing AI-driven detection systems and indicates the importance of integrating diverse data sources for enhanced clinical decision-making.