Multi-Scale and Multi-Modal Contrastive Learning for Hyperspectral and LiDAR Classification

The combination of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data enables the simultaneous exploitation of rich spectral and elevation information and has become a hot research topic in land-cover classification. However, the reliance on large datasets with costly manual annotations presents a significant challenge to the scalability of models. To address the issues of labeled data scarcity and model generalization, we propose a dual-branch multi-scale contrastive learning (DMCL) framework for the joint classification of HSI and LiDAR data. The model training process consists of self-supervised pre-training and supervised fine-tuning. During the pre-training phase, intra-modal and inter-modal contrastive tasks are exploited to learn multi-modal invariant features and capture the correlations between different modalities. In the fine-tuning stage, a simple yet effective fusion module is introduced to obtain more effective representations and adapt the model for the joint classification task. Experimental results show that, compared with other methods, the DMCL framework achieves the highest overall accuracy of 88.68%, 98.09%, and 76.25% on the Houston 2013, Trento, and MUUFL datasets, respectively.