DSU-Net：An Improved U-Net Model Based on DINOv2 and SAM2 with Multi-scale Cross-model Feature Enhancement

This paper addresses two critical challenges in applying general image segmentation models to specific domains: high training costs and inadequate feature representation. We propose a novel multi-scale feature collaboration framework that leverages DINOv2 and SAM2. Our key innovations are threefold: First, we propose a multi-level feature collaboration mechanism that integrates the DINOv2 with SAM2, enabling effective multi-scale feature fusion enhanced by high-dimensional semantics. Next we design a lightweight adapter that operates under parameter-freezing conditions, complemented by a cross-layer feature interaction module that facilitates efficient domain knowledge transfer. Finally, we develop a U-shaped network based on U-Net, that performs multi-granularity feature aggregation decoding through spatial feature fusion, and mitigates multi-level accuracy loss using a weighted summation of diverse layer loss functions. Experimental results demonstrate that our method significantly outperforms existing state-of-the-art techniques in Salient Object Detection (SOD) and Camouflaged Object Detection (COD), achieving improvements of 1.25% and 1.02%, respectively. Notably, our approach requires training only 1.38% of the parameters to adapt to various downstream image segmentation tasks, eliminating the need for fine-tuning base model. This framework effectively overcomes the adaptation challenges faced by general segmentation models in specialized contexts, providing a robust solution for image segmentation in critical fields such as medical image analysis and industrial inspection. Project page: https://github.com/CheneyXuYiMin/SAM2DINO-Seg.