Cross-modal Frequency-aware Transformer for Multimodal Medical Image Fusion

Fusing multimodal medical images seeks to amalgamate their unique informational strengths, thereby enhancing the precision of clinical diagnostics and the accuracy of surgical navigation. The resulting image quality in multimodal fusion is contingent upon the characteristics derived from individual modalities and the strategies employed for integrating this diverse data. Nevertheless, current deep learning-driven fusion techniques, although adept at harnessing the semantic aspects of each imaging mode, struggle to differentiate between the beneficial low-frequency and high-frequency details within each modality. This limitation hinders their ability to capture comprehensive global insights and to achieve dynamic, context-aware fusion. To address these issues, we introduce an innovative fusion framework that utilizes Fourier transforms and incorporates a cross-modal frequency-aware transformer. This framework includes a frequency-aware cross-attention mechanism for adaptive feature fusion across modalities by exchanging keys and queries between spatial and frequency domains. It also computes cross-attention scores to facilitate the integration of spatial and frequency data. Additionally, we employ a hybrid approach of gradient preservation and transformers for feature extraction, capturing both local specifics and global information. Our model surpasses existing methods assessments, markedly improving fused image quality.