ProMM-RS: Exploring Probabilistic Learning for Multi-Modal Remote Sensing Image Representations

Remote sensing imagery offers diverse modalities, such as synthetic aperture radar and multispectral data, which can bring rich, complementary and valuable information about observed scenes. This information is of paramount importance for downstream applications (e.g. land cover mapping, natural resources monitoring, human settlement characterization) that may benefit from such complementarity. Remote sensing imagery often suffers from a lack of labeled data which can hamper the learning of good representations via state-of-the-art supervised methods. Self-supervised learning has thus emerged as a promising paradigm for remote sensing feature extraction, enabling the extraction of meaningful features without reliance on labeled data. While existing multimodal contrastive models effectively capture shared information between modalities, they often struggle to account for the inherent heterogeneity of multimodal re-mote sensing data. This limitation prevents them from fully leveraging the complementarity of multimodal remote sensing data. Probabilistic representation learning has emerged as a powerful approach to capture the inherent uncertainty and diversity in multimodal relationships. In this paper we present ProMM-RS, a novel multimodal self-supervised training framework incorporating a joint probabilistic embedding space to explicitly model the uncertainty of representations between different inputs and modalities. We evaluate our learned representations with a scene classification downstream task from Sentinel optical and radar images, effectively showing the potential of probabilistic embeddings as a way to measure the relevancy of each modality representation, especially under an obstructed dataset.