Multimodal Clinical Decision Support for Melanoma Diagnosis Using Retrieval-Augmented Generation and Vision-Language Models

Malignant melanoma is a lethal form of skin cancer, where early and precise diagnosis is essential for improving survival. While convolutional neural networks (CNNs) have shown promise in dermoscopic image analysis, they often neglect patient metadata that clinicians routinely consider. Vision-language models (VLMs) provide a unified architecture for multimodal understanding; however, their utility in clinical domains remains limited due to domain mismatch and lack of adaptation. To address these limitations, we propose a retrieval-augmented framework that integrates pre-trained VLMs with a multimodal vector database of dermoscopic images and patient metadata. For each input, the system retrieves semantically similar cases and inserts them into structured prompts, which enable few-shot classification without fine-tuning. Our method outperforms image-based, text-based, and embedding-fusion baselines by yielding higher F1-scores and reducing both false positives and false negatives. Our findings support retrieval-augmented prompting as a reliable and scalable approach to clinical decision support in melanoma diagnosis.