Global-Local Medical SAM Adaptor Based on Full Adaption

Visual language models (VLM) have made great breakthroughs in natural image segmentation, and they show consistently high-quality zero-shot transferring capability in many down-stream tasks with prompt engineering. The help of modern parameter-efficient fine-tuning techniques, such as low-rank adaption (LoRA) and Medical SAM adaptor (Med-SA) further enables VLM to be well transferred to medical domains. Regardless of the performance, these methods choose to fine-tune VLM in a local manner, where adapters are specified to adapt some layers vital to the down-stream tasks, making them separated by frozen layers in forward path and lack of collaboration in back-propagation. To resolve this problem, we present a global adaption technique and create a novel global medical SAM adaptor (GMed-SA), which can adapt SAM in a collaborative adaption manner. Specifically, all adapters communicate via fully connected layers as a unified sub-network, enabling them to work collaboratively in fine-tuning. We further combine GMed-SA and Med-SA to propose a global-local medical SAM adaptor (GL-Med-SA) for 2D melanoma segmentation dataset. The extensive experiments show that the combined method GL-Med-SA slightly improve the segmentation performance on the challenging publicly available 2D melanoma dataset while only enlarging minor trainable weight parameters and outperforming several state-of-the-art methods in both Dice metrics and Intersection over union (IoU), which demonstrates its superiority.