Vision-Language Models for Skin Disease Image-to-Text Retrieval Using Visual and Textual Understanding in Dermatology

Automated skin disease analysis can enhance diagnostic accuracy and clinical workflow efficiency. Traditional Convolutional Neural Networks excel at unimodal image classification but neglect textual clinical context. This work presents a Vision-Language Model approach for skin disease image-to-text retrieval that addresses limitations of unimodal and task-specific multimodal methods. We fine-tune the CLIP model using ViT-B architecture on SkinCAP, a dermatology dataset containing expert-curated medical captions. The model optimizes semantic alignment between skin lesion images and textual descriptions through contrastive learning. Performance evaluation employs Recall at K, Mean Rank, and Mean Reciprocal Rank metrics. Results on SkinCAP test data show Recall at 1 of 68.5 percent, Mean Rank of 3.2, and Mean Reciprocal Rank of 0.64. The correct medical caption ranks first in over 68 percent of retrievals. These results demonstrate effective fine-tuning of CLIP for dermatological applications and establish the capability of Vision-Language Models to integrate visual and textual understanding in medical imaging. The approach contributes to multimodal retrieval systems for clinical dermatology applications.