Dual-Prototype Learning With Multisemantic Fusion for Cross-Domain Few-Shot Hyperspectral Image Classification

Multimodal learning has demonstrated outstanding performance in various fields, including visual tasks, due to its ability to integrate information from different data sources. In the field of hyperspectral image classification, although existing research can handle complex image data, the utilization of semantic information is insufficient. Over-reliance on a single image modality overlooks the synergistic effects of cross-modal information, thereby limiting the model’s performance in category center representation and discrimination, especially in scenarios with scarce samples. To address these challenges, this paper proposes a Dual-Prototype Learning with Multi-Semantic Fusion method (DPL-MSF). Specifically, within the DPL-MSF framework, precise text prototypes are constructed by leveraging category textual information and prompt learning strategies to capture the first layer of semantic information. Additionally, soft labels are introduced into the image prototypes, and a feature fusion module is designed to deeply integrate soft label information, thereby obtaining another layer of semantic information. The primary role of the text prototype is to assist in the generation of image prototypes before the introduction of soft labels, through contrastive learning methods. Under the synergistic effect of multiple semantic information, the final class-level prototypes generated are more representative and discriminative, significantly enhancing the performance of few-shot learning. Experiments conducted on multiple standard hyperspectral image datasets have shown that DPL-MSF has significant advantages in scenarios with scarce samples and cross-domain settings. The code will be available at https://github.com/AIYAU/DPL-MSF.