Semi-supervised multimodal contrastive regularization network for remote sensing hyperspectral and LiDAR classification

Semi-supervised learning networks based on pseudo-label method have been widely applied to multimodal image classification tasks to improve model accuracy when labeled data are scarce. However, it remains a critical challenge for multimodal data to generate high-quality pseudo-labels and effectively leverage unreliable unlabeled samples. Facing the challenge, a semi-supervised multimodal contrastive regularization network (SMCRN) for hyperspectral and LiDAR classification is proposed to fully exploit unlabeled samples. Firstly, to generate rational pseudo-labels for all unlabeled samples, we design a multimodal label assignment strategy (MLAS) to establish a training sample set through cross-modal comparison of superpixel features. Then, substantial class-balance samples are fed into a three-stream semi-supervised network to execute the class prediction, in which the unlabeled samples are divided into unreliable and reliable subsets according to the predicted class probabilities. For the reliably predicted samples, consistency regularization enforces the prediction invariance in an identical sample across different streams for enhancing the model robustness, while the unreliable samples are utilized in contrastive learning by providing their negative features, thereby improving the discriminative ability. Finally, three loss functions are integrated to optimize the network. Extensive experiments on three multimodal remote sensing datasets demonstrate that the proposed SMCRN achieves superior classification accuracy compared to state-of-the-art methods.