MVFG: Enhancing Semantic Segmentation via Mix Vision Transformer With a Feature Guidance Module Powered by SAM Masks

The challenges in semantic segmentation of remote sensing images arise primarily from significant variations in target sizes, diverse shapes, and complex background environments. To address these challenges, we have developed a feature-guided semantic segmentation model for remote sensing images, further enhancing its accuracy with high-fidelity masks generated by the SAM vision large model. The principal contributions of this study are outlined as follows: We introduce a semantic segmentation model that employs feature guidance, ingeniously integrating CNN and Transformer architectures to effectively capture both the local details and the global context of remote sensing images. We propose an improved method based on the SAM vision large model, featuring a dual-branch structure that incorporates two key modules: semantic voting and mask refinement. The semantic voting module corrects semantic errors in both branches. Subsequently, a mask refinement module is utilized to further enhance these masks. In experimental validations using the iSAID dataset, the model significantly improved the mean Intersection over Union (mIoU) to 61.54% and the mean <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</i>1 score to 74.69%, underscoring the significance and applicability of the proposed methodology.