A Holistic Approach to Multi-Modal Skin Lesion Diagnosis Supported by Statistical and Explainability-Based Investigation of Artifacts

Abstract Skin cancer is a prevalent and potentially life-threatening condition, where early and accurate detection is crucial for effective treatment. Traditional diagnosis, based on visual assessment, faces challenges due to inter-class variability and the similarity between benign and malignant lesions. Although automated diagnostic systems are gaining attention, few offer comprehensive evaluations. This study addresses this gap by systematically assessing such systems and reporting extensive performance metrics. This study contributes to both artificial intelligence and dermatological image analysis by introducing an automated system for multi-class skin lesion classification. The system was evaluated under various configurations, including original and artifact-inpainted images. Three approaches were employed: deep learning, radiomics-based and hybrid. Additionally, to enhance the interpretability, the system generates segmentation masks and attribution maps. Models were evaluated using five-fold cross-validation on the International Skin Imaging Collaboration (ISIC) 2018 Challenge Task 3 training dataset, with final performance assessed on the official test set. The best-performing model achieved a micro accuracy of 79.76±0.73, macro F1-score of 73.62±1.38, precision of 74.80±0.84, and recall of 73.93±2.19, surpassing comparable methods reported in the literature. The contribution to the research community is the release of a novel set of 57630 binary artifact labels for the ISIC-2018 Task 3 dataset, enabling research on the impact of visual artifacts such as hair, rulers, and interface fluids. This contribution is supported by statistical analysis and attribution map assessment, revealing significant differences in artifact occurrence across lesion classes and highlighting the utility of these labels in identifying and mitigating bias in skin lesion classification.