Evaluating the Feasibility of Vision-Language Models in Skin Cancer Detection: A Comparative Study with CNNs and Vision Transformers

This study presents a comprehensive evaluation of vision-language models (VLMs) for skin lesion classification, comparing them with conventional convolutional neural networks (CNNs) and Vision Transformer (ViT) architectures. Using the HAM10000 dataset, we evaluated six models in four key dimensions: classification accuracy, robustness to visual perturbations, zero-shot generalization to unseen lesion types, and the quality of semantic explanations. Although traditional vision-only models achieve higher accuracy in clean images, their performance degrades significantly under various types of image distortion. In contrast, VLMs—particularly Qwen2.5—demonstrate stronger robustness and generate more coherent and clinically relevant explanations. However, they still fail to achieve overall classification performance and exhibit limited generalization in zero-shot settings. These findings highlight the trade-offs between task-specific accuracy and multimodal adaptability, offering practical insights into the current capabilities and limitations of VLMs in dermatology-focused artificial intelligence applications.