Robust and Efficient Hybrid Transformer-Convolutional Network for Enhanced Skin Lesion Segmentation and Classification

Skin lesion analysis is crucial for the early detection and treatment of skin cancers and demands the precise and reliable segmentation and classification models. While existing deep learning models are promising in this area, they have limitations in handling the complex skin lesions and achieving high accuracy and generalizability across diverse datasets. This paper proposes a novel hybrid model, HLS-Net, which integrates transformer and Convolutional Neural Network (CNN) architectures to address these challenges. The model is employed with a dual-encoder structure, which combines the lightweight CNN branch (e.g. MobileNetV3) for efficient local feature extraction and a Vision Transformer (ViT) branch to capture the long-range dependencies and global contextual information. A hybrid decoder is designed with transposed convolution, upsampling, and channel weighting techniques to refine the feature maps and to improve the segmentation accuracy. Additionally, the model is advanced using deep supervision with Monte Carlo Dropout for uncertainty estimation to ensure the robust predictions even in challenging scenarios. A dedicated classification head is designed with a multi-layer perceptron to support the multi-task learning and to improve the diagnostic accuracy. Extensive experiments performed on the ISIC 2017, ISIC 2018, and PH2 datasets depict the model’s superiority over the selected baseline models. HLS-Net achieved the DCE of 97.20% and IoU of 96.81% on the PH2 dataset, which is outperforming the baseline models. For classification, it attains an accuracy of 97.25% on PH2, 95.06% on ISIC 2017, and 94.78% on ISIC 2018. The extensive results obtained from the experiments highlight HLS-Net’s efficiency, robustness, and generalizability, making it a valuable tool for clinical skin lesion analysis.