An efficient fine tuning strategy of segment anything model for polyp segmentation

Colon cancer is a prevalent disease on a global scale, thus making its detection and prevention a critical area in the medical field. In addressing the challenges of high annotation costs and the need for improved accuracy in colon polyp detection, this study explores the segment anything model (SAM) application and fine-tuning strategies for colon polyp segmentation. Conventional full fine-tuning approaches frequently result in catastrophic forgetting, thereby compromising the model's generalization capabilities. To address this challenge, this paper proposes an efficient fine-tuning method, PSF-SAM, which mitigates catastrophic forgetting while enhancing performance in few-shot scenarios. This is achieved by freezing most SAM parameters and optimizing only specific structures. The efficacy of PSF-SAM is substantiated by experimental evaluations on the Kvasir-SEG and CVC-ClinicDB datasets, which demonstrate its superior performance in metrics such as mDice coefficients and mIoU, as well as its notable advantages in few-shot learning scenarios when compared to existing fine-tuning methods.