Multimodal Image Classification Based on Convolutional Network and Attention-Based Hidden Markov Random Field

In this paper, a multimodal deep architecture for classification of light detection and ranging (LiDAR) and hyperspectral image (HSI) is proposed, acquiring the knowledge of both modalities by leveraging modality specific information as well as their complementary information. The proposed model consists of two main steps. First, to improve the performance of a two-dimensional convolutional neural network (2DCNN), low-frequency features with maximum autocorrelation factor of HSI are injected into 2DCNN which are called multi-scale features of 2DCNN. Second, to improve the accuracy of 2DCNN and extract smooth and semantic information, the posterior energy of hidden Markov random field (HMRF) is modified by using Gaussian attention and albedo recovery attention mechanisms and energies of LiDAR and HMRF. Then, these features are fused based on another attention mechanism called attention-based HMRF. Moreover, this HMRF model is used for fusion of HSI and LiDAR. The proposed model is tested on the Houston 2013, Trento and MUUFL datasets and compared with several state-of-the-art methods. The resulting classification accuracies through ablation study show the superior performance of the proposed method.