MemDDNet: Memory-driven two-branch two-stage CT-MRI image fusion network

Professionals still struggle to identify illnesses from pictures taken with a single imaging modality. Image fusion can merge images from multiple imaging systems on a specific organ. The fusion of structural and functional information will make diagnosis easier. Our work proposes a network called MemDDNet, whose main task is to fuse CT-MRI images of the brain. MemDDNet is a two-branch feature decomposition network with memory-correlation-driven. MemDDNet extracts the base and detail features of CT and MRI images respectively through a two-branch feature decomposition network. The fused images are obtained by feeding the extracted features into a trained feature fusion layer. Meanwhile, the loss function driven by memory and correlation enriches the supervised signal. Quantitative comparisons show that our network achieves optimality or sub-optimality on all metrics. Especially in SF, MI, and QBA/F, there have been improvements of 19.57%, 8.12% and 13.56%, respectively. Qualitative comparison shows that the fused images generated by our network are more in line with the doctor's needs when reading films. We have validated our network through various experiments, and the results have shown the effectiveness of our network in CT-MRI image fusion tasks, especially in solving the problem of ground truth output deficiency to a certain extent.