ATIS-Driven 3DCNet: A Novel Three-Stream Hyperspectral Fusion Framework with Knowledge from Downstream Classification Performance

Reconstructing high-resolution hyperspectral images (HR-HSIs) by fusing low-resolution hyperspectral images (LR-HSIs) and high-resolution multispectral images (HR-MSIs) is a significant challenge in image processing. Traditional fusion methods focus on visual and statistical metrics, often neglecting the requirements of downstream tasks. To address this gap, we propose a novel three-stream fusion network, 3DCNet, designed to integrate spatial and spectral information from LR-HSIs and HR-MSIs. The framework includes two dedicated branches for extracting spatial and spectral features, alongside a hybrid spatial–spectral branch (HSSI). The spatial block (SpatB) and the spectral block (SpecB) are designed to extract spatial and spectral details. The training process employs the global loss, spatial edge loss, and spectral angle loss for fusion tasks, with an alternating training iteration strategy (ATIS) to enhance downstream classification by iteratively refining the fusion and classification networks. Fusion experiments on seven datasets demonstrate that 3DCNet outperforms existing methods in generating high-quality HR-HSIs. Superior performance in downstream classification tasks on four datasets proves the importance of the ATIS. Ablation studies validate the importance of each module and the ATIS process. The 3DCNet framework not only advances the fusion process by leveraging downstream knowledge but also sets a new benchmark for classification-oriented hyperspectral fusion.