An interpretable cross-attentive multi-modal MRI fusion framework for schizophrenia identification

Functional MRI (fMRI) and structural MRI (sMRI) offer complementary insights into brain function and anatomy, but their integration for schizophrenia identification remains challenging due to modality heterogeneity. Many existing methods fall short of effective modeling of the interaction between two modalities. We propose CAMF, a Cross-Attentive Multi-modal Fusion framework that employs self-attention to capture intra-modal patterns and cross-attention to learn inter-modal relationships. In addition, we introduce the gradient-guided score-class activation map to enhance interpretability by highlighting salient features. Our approach significantly improves the accuracy in classifying schizophrenia, as demonstrated by the evaluation of multi-modal brain imaging datasets from four cohorts of schizophrenia studies. Furthermore, the model identifies functional networks and anatomical regions aligned with established biomarkers. CAMF provides an accurate and interpretable framework for multimodal brain imaging analysis, offering new insights into schizophrenia-related alterations. • Unique attention-based data integration method to fuse fMRI and sMRI data for schizophrenia identification with better classification performance. • Interpretable method for deep learning models to highlight key brain regions related to schizophrenia. • The key regions identified by fMRI and sMRI respectively show spatial alignment in human brain, in concordance with the results reported in prior studies.