<scp>MAMSN</scp>: Multi‐Attention Interaction and Multi‐Scale Fusion Network for Spectral Reconstruction From <scp>RGB</scp> Images

ABSTRACT In the present era, hyperspectral images have become a pervasive tool in a multitude of fields. In order to provide a feasible alternative for scenarios where hyperspectral imaging equipment is not accessible, numerous researchers have endeavored to reconstruct hyperspectral information from limited spectral measurements, leading to the development of spectral reconstruction (SR) algorithms that primarily focus on the visible spectrum. In light of the remarkable advancements achieved in many computer vision tasks through the application of deep learning, an increasing number of SR works aim to leverage deeper and wider convolutional neural networks (CNNs) to learn intricate mappings of SR. However, the majority of deep learning methods tend to neglect the design of initial up‐sampling when constructing networks. While some methods introduce innovative attention mechanisms, their transferability is limited, impeding further improvement in SR accuracy. To address these issues, we propose a multi‐attention interaction and multi‐scale fusion network (MAMSN) for SR. It employs a shunt‐confluence multi‐branch architecture to learn multi‐scale information in images. Furthermore, we have devised a separable enhanced up‐sampling (SEU) module, situated at the network head, which processes spatial and channel information separately to produce more refined initial up‐sampling results. To fully extract features at different scales for visible‐spectrum spectral reconstruction, we introduce an adaptive enhanced channel attention (AECA) mechanism and a joint complementary multi‐head self‐attention (JCMS) mechanism, which are combined into a more powerful feature extraction module, the dual residual double attention block (DRDAB), through a dual residual structure. The experimental results show that the proposed MAMSN network outperforms other SR methods in overall performance, particularly in quantitative metrics and perceptual quality.