Comparative Analysis of Single‐Stream and Multi‐Stream Models for Medical Image Fusion

ABSTRACT Medical image fusion supports more informative image interpretation by integrating complementary information from different imaging modalities. This paper presents a controlled comparative study of three input‐stream configurations for deep learning‐based multi‐modal medical image fusion on Positron Emission Tomography‐Magnetic Resonance Imaging (PET‐MRI) and Single‐Photon Emission Computed Tomography‐Magnetic Resonance Imaging (SPECT‐MRI) fusion tasks. The evaluated configurations include (1) Single‐Input, which fuses the two modalities at the input level; (2) Dual‐Input, which processes each modality through a separate encoder branch before feature fusion; and (3) Tri‐Input, which combines individual modality streams with an additional concatenated input stream to learn both modality‐specific and joint representations. All configurations use the same U‐Net‐based encoder‐decoder backbone with Transformer‐CNN modules, allowing the effect of input‐stream design to be examined under a consistent architectural and training setting. Performance is evaluated using entropy (EN), mutual information (MI), feature mutual information (FMI), correlation coefficient (CC), peak signal‐to‐noise ratio (PSNR), structural similarity index measure (SSIM), validation loss, training time per epoch, external baseline comparison, and statistical significance analysis. The results show that Dual‐Input provides the most balanced overall performance, especially for PET‐MRI fusion, while Tri‐Input achieves stronger feature‐transfer performance on selected SPECT‐MRI metrics. Single‐Input offers the fastest training speed but lower overall fusion quality among the evaluated stream configurations. These findings clarify the trade‐offs among early fusion, separate modality processing, and hybrid multi‐stream input design for functional‐anatomical medical image fusion, while remaining limited to the evaluated dataset, modality pairs, and image‐quality metrics.