Vision Transformer-Driven LiDAR Data Fusion for Enhanced Hyperspectral Image Classification

Multimodal fusion using Hyperspectral (HS) and Light Detection and Ranging (LiDAR) data has become a pivotal research focus in remote sensing and geospatial analysis. Vision Transformers (ViTs) have emerged as a transformative technology, offering sophisticated multimodal feature extraction and data fusion capabilities. This study investigates two previously unexplored multimodal fusion strategies for land cover classification tasks within the ViT architecture: self-attention-based and cross-attention-based fusion. These strategies aim to enhance the classification performance by utilizing shared and modality-specific parameters. Extensive experiments on benchmark datasets from the University of Houston demonstrate that the proposed cross-attention fusion method with ViTs outperforms the baseline ViT, previously proposed ViT-based fusion methods, and self-attention-based fusion approaches. Further, we demonstrate that late fusion allows for better learning of modality-specific features separately compared to early fusion. The trained models and code are available on GitHub.