HiTrans-SAM: Hierarchical Transformer Encoder and SAM-Augmented Inputs for Multi-Scale Remote Sensing Image Segmentation

Semantic segmentation of remote sensing images is challenging due to complex scenes, substantial variations in object scales, and ambiguous boundaries. In this study, we propose a novel method, HiTrans-SAM: Hierarchical Transformer Encoder and SAM-Augmented Inputs for Multi-Scale Remote Sensing Image Segmentation. The framework adopts an encoder-decoder architecture. First, prior to encoding, the input image is enhanced using SAM to incorporate boundary prior maps generated by SAM, thereby mitigating boundary ambiguity. Subsequently, a Hierarchical Transformer Encoder is integrated into the encoding network to facilitate information propagation. This module captures high-resolution spatial details while effectively leveraging global contextual relationships. During the decoding phase, multi-scale feature fusion is performed to ensure comprehensive utilization of features across varying scales, ultimately improving segmentation accuracy. Experiments on the LoveDA and Potsdam datasets demonstrate state-of-the-art performance, achieving mean Intersection over Union (mIoU) values of 53.52% (LoveDA), 79.45% (Potsdam) and 75.12%(Vaihingen), significantly outperforming existing methods. The results validate the algorithm’s efficacy in enhancing segmentation accuracy through boundary refinement, context modeling, and multi-scale feature fusion.