A Progressive Registration-Fusion Co-Optimization A-Mamba Network: Toward Deep Unregistered Hyperspectral and Multispectral Fusion

The existing methods of hyperspectral image (HSI) and multispectral image (MSI) fusion usually overlook the fact that multi-source images acquired under different imaging conditions are generally not perfectly registered. Despite the many such methods that have begun to address registration issues, it is still a challenge that most works perform registration and fusion as two separate steps, resulting in a cumulative error. To address this challenge, we propose a progressive registration-fusion co-optimization A-Mamba network (PRFCoAM), which iteratively optimizes the modal-aligned progressively registration-fusion (MAPRF) module to adaptively corrects the deformation from an extensive to a detailed level and refines the fusion results at each level to achieve progressive registration-fusion co-optimization. The proposed MAPRF module integrates the modal unified local aware registration (MULAR) block and interactive attention Mamba fusion (IAMF) block, which facilitates the network comprehensively and efficiently capture features of different levels. Specifically, MULAR adaptively learns spectral and spatial degradation functions to transform the input images into a unified modality and progressively repairs non-rigid pixel offsets by capturing the correlations and differences between corresponding regions of images. IAMF multi-directionally scans the spatial and spectral global dependent features of the well-registered images, which can stimulate the potential of Mamba in fusion and achieve a win-win situation of computational efficiency and selectivity advantages in the global acceptance domain. Extensive experiments demonstrate PRFCoAM can flexibly deal with different degrees and kinds of non-rigid deformation and achieves state-of-the-art performance. The code will be available at https://github.com/Jiahuiqu/PRFCoAM-for-HSI-MSI-Registration-Fusion.