DermFusion: an uncertainty-aware cross-attention CNN–Transformer framework for multi-task skin lesion classification and segmentation

Melanoma is the deadliest form of skin cancer, yet early-stage disease is nearly always curable, making accurate automated detection a high-value clinical target. State-of-the-art deep-learning systems for dermoscopy analysis typically treat classification and segmentation as separate tasks and produce insufficient confidence scores, potentially limiting their safe deployment. We proposed DermFusion , which integrates both within a single ResNet-50+CBAM/ViT-B/16 dual-backbone network connected by a bidirectional cross-attention fusion layer. A U-Net decoder with Focal-Tversky deep supervision handles boundary prediction, while a Monte Carlo dropout head quantifies predictive uncertainty. Post-hoc temperature scaling calibrates model confidence, ensuring predicted probabilities better match actual accuracy. On the HAM10000 dataset, DermFusion achieves 87.82% accuracy, 0.9799 ROC-AUC, 0.9359 Dice, and 91.2% melanoma sensitivity. Calibration improves significantly, with the expected calibration error decreasing from 0.4395 to 0.1575, representing a 64.1% relative reduction. It has been observed that when an uncertainty-based referral threshold applies, 68% of cases classified autonomously reach 96.4% accuracy, with the remaining 32% flagged for expert review. external robustness evaluation on PH \(^{2}\) , ISIC 2019, and PAD-UFES-20 datasets was applied, which shows that the combined representation works well across different acquisition domains. Ablation, fairness, and failure-mode studies characterize where and why the proposed DermFusion system succeeds or breaks down.