GeoMM-SSL: Integrating Geospatial Object Relations in Multimodal Self-Supervised Learning for Semantic Segmentation of Remote Sensing Images

Self-Supervised Learning (SSL) has emerged as a promising approach for pre-training tasks by learning latent task-agnostic representations without labels. Currently, the pretrained SSL models for semantic segmentation of remote sensing images have attracted increasing attention. However, current pre-trained SSL models tend to focus solely on either global semantics or local spatial representation. Additionally, existing pre-trained SSL models ignore the spatial relationship among various objects, which could help infer co-occurrence between geospatial objects. In this paper, we propose a multi-modal pretrained SSL (GeoMM-SSL) framework that explicitly integrates geospatial object relations. The proposed framework includes a teacher-student framework with residual gated guidable attention units (GAU) as the backbone, a multi-head graph attention network (MGAT) that encodes prior knowledge of geospatial object relations, a multi-modal representation fusion (MRF) module that facilitates mutual learning between visual features of remote sensing images and topological features of geospatial object relations, and a multi-level loss function that performs multiple levels of evaluation, enabling the model to learn the data representation at the pixel, object, and global levels. We conducted comprehensive experiments to compare GeoMM-SSL with 13 existing SSL methods on 10 public semantic segmentation datasets, and the results show that GeoMM-SSL achieves optimal results.