Injecting Visual Prototypes into CLIP-driven Long-Tailed Skin Lesion Classification

Skin lesion classification plays a vital role for the early and accurate diagnosis of dermatology. Traditional methods rely on subjective clinical expertise but remain challenged by long-tailed data distributions and insufficient utilization of visual information. With the advancement of artificial intelligence, an increasing number of computer-aided approaches have been attempted for skin lesion classification tasks. In this paper, we propose DermSense, a novel framework to enhance skin lesion classification by integrating visual prototypes, LLM-generated fine-grained attributes with the CLIP model. Leveraging a dual-matching strategy and a cross-modal fusion module enriches visual textual feature representations without the need for fine-tuning the CLIP backbone network, and to significantly reduce computational overheads. Additionally, we employed a Logits-Adjusted loss function to address the class imbalance. Extensive experiments on ISIC2018 and Derm7pt datasets demonstrate that DermSense significantly outperforms state-of-the-art methods in balanced accuracy, particularly for melanoma classification. Our results highlight the potential of DermSense to advance skin disease diagnosis while maintaining scalability and efficiency.