PSAM: Prompt-based Segment Anything Model Adaption for Medical Image Segmentation

In the current landscape where large models are increasingly becoming the norm for task solving, maximizing the utilization of these models has emerged as a focal point of research. The Segment Anything Model (SAM), an eminent large-scale image segmentation model using a new task, model, and dataset, has garnered recognition for its efficacy across different scenarios. However, the effectiveness of SAM is hindered in the medical domain due to the scarcity of available medical images, leading to suboptimal training and inadequate adaptation of its feature extractor to medical imagery. In this work, we propose PSAM, which built upon SAM to explore a new research paradigm of customizing large-scale models to meet the demands of medical image segmentation tasks. This is achieved through a two-fold strategy: Firstly, we incorporate parallel feature extraction branches into SAM, guided by task-specific prompts derived from CLIP, enhancing its ability to extract relevant features. Secondly, we introduce an enhanced, visually task-friendly adapter mechanism, which effectively injects medical knowledge into SAM's ViT image encoder for facilitating adaptive task execution in medical image scenarios. Our experimental findings demonstrate the effectiveness of PSAM in accurately segmenting medical images, underscoring its potential as a valuable tool in the medical imaging domain.