Hybrid Transformer-Based Framework for Building Detection in High-Resolution Satellite Imagery: Integrating SAM, PVT, and Mask2Former

Building extraction from high-resolution satellite imagery is important for urban planning, infrastructure monitoring, disaster management, and other applications. In this paper, we perform a comparative and integrative study of these transformer-based architectures for automated building segmentation on the challenging INRIA Aerial Image Labeling Dataset and WHU Building Change Detection Dataset. The effectiveness of three state-of-the-art models, the Segment Anything Model (SAM), the Pyramid Vision Transformer (PVT), and Mask2Former, are evaluated individually and in integrated settings. Segment Anything Model (SAM), a model based on Vision Transformer (ViT) architecture, is a unified model for zero-shot, prompt-based segmentation capable of generalizing well to the different visual domains. PVT introduces a hybrid architecture for multi-scale feature extraction, so it is the better option for dense prediction of high-resolution images. Mask2Former, as a unified segmentation model, constructs high-referring context-aware detail segmentation masks by refining object queries through masked cross-attention. In CPU-restrained environments, we tested four configurations: SAM with CNN, SAM with PVT, Mask2Former with PVT, and hybrid integration of SAM, PVT, and Mask2Former. In terms of standard metrics, including precision, recall, F1-score, and mIoU, hybrid configuration significantly outperforms the other configurations, yielding an overall F1-score of 0.885 and exhibiting greater robustness in complex urban scenes containing occlusions, architectural diversity, and building scale variability. Although the hybrid model adds a bit of computational overhead, it benefits from these aspects: first, prompt generalization of SAM; second, spatial feature extraction from PVT; and third, mask refinements from Mask2Former. This approach can be especially useful for applications that prioritize segmentation accuracy over processing speed. The results of this study encourage the adaptation of multi-transformer hybrid architectures for scalable and accurate building detection in remote sensing applications with a mission-oriented focus on processing speed. The results of this study encourage the adaptation of multi-transformer hybrid architectures for scalable and accurate building detection in remote sensing applications with a mission-oriented focus. The ablation studies inferred the promising performance of the proposed integrated approach.