MoVL: Exploring Fusion Strategies for the Domain-Adaptive Application of Pretrained Models in Medical Imaging Tasks

Medical images are often more difficult to acquire than natural images due to the specialized equipment and technology required, leading to fewer available medical image datasets. This limitation poses challenges in training robust pretrained medical vision models. How to best leverage natural pretrained vision models and adapt them to the medical domain remains an open question. For image classification, linear probing (LP) is a commonly used technique. However, LP primarily focuses on the output after feature extraction, without addressing the inherent differences between medical images and natural image-based pretrained models. To bridge this gap, we introduce visual prompting (VP) and investigate strategies for integrating LP and VP. We propose a joint learning framework with a composite loss function that includes a categorization loss and a discrepancy loss, capturing the variance between prompted and plain images, naming this joint training strategy MoVL (Mixture of Visual Prompting and Linear Probe). We experiment on four medical image classification datasets, with two mainstream architectures, ResNet and CLIP. Results shows that without changing the parameters and architecture of backbone model and with less parameters, there is potential for MoVL to achieve full finetune (FF) accuracy (on four medical datasets, average 90.91% for MoVL and 91.13% for FF). On out of distribution medical dataset, our method (90.33%) can outperform FF (85.15%) with absolute 5.18 % lead.