BMoE-AdaLoRA+: Parameter Efficient Fine-tuning for Segmenting Anything Model

Segment Anything Model (SAM), as a fundamental framework in the field of image segmentation, demonstrates excellent zero-shot capabilities in typical scenes. However, its advantages are diminished when it is applied to specialized fileds such as medical images and remote sensing, and Parameter Efficient Fine-Tuning (PEFT) is a key way to solve this problem. On the basis of the pre-trained model, it provides a large amount of professional domain knowledge for the model and improves its domain-specific segmentation ability. However, we find that existing parameter efficient fine-tuning methods suffer from insufficient robustness, and the addition of multi-category domain data makes the model inevitably have the problem of inter-task interference. To address this challenge, we propose BMoE-AdaLora+. Firstly, we propose the concept of local smoothed sensitivity to enable the model to obtain stronger robustness during the training process. Secondly, we integrate the AdaLora+ module as an expert into the Balanced Mixture-of-Experts (BMoE), a MoE-style plug-in designed to motivate each expert to focus on a single type of data, which is used to minimize inter-task interference in multi-category fine-tuning by balancing the utilization of all experts. Experimental results show that BMoE-AdaLora+ can significantly improve the segmentation ability of the model in multi-category professional fields with the increase of domain knowledge data.