Multi-Dimension-Embedding-Aware Modality Fusion Transformer for Psychiatric Disorder Diagnosis

Deep learning and neuroimaging techniques are important in diagnosing psychiatric disorders. However, previous studies on psychiatric disorders classification mainly focus on using functional connectivity matrices of resting-state functional magnetic resonance imaging(rs-fMRI) as input. This approach fails to fully utilize the rich temporal information present in rs-fMRI data. Additionally, deep learning models equipped with effective modality fusion strategies have the potential to better utilize multi-modality data, thereby enhancing model performance. In this work, we proposed a multi-dimension-embedding-aware modality fusion transformer (MFFormer) for diagnosing schizophrenia and bipolar disorder using rs-fMRI and T1 weighted structural MRI (T1w sMRI). Concretely, to fully utilize the temporal information of rs-fMRI and spatial information of sMRI, we constructed a deep learning architecture that takes as input 2D time series of rs-fMRI and 3D volumes T1w. Furthermore, to promote intra-modality attention and information fusion across different modalities, a fusion transformer module (FTM) is designed through extensive self-attention of hybrid feature maps of multi-modality. In addition, a dimension-up and dimension-down strategy is suggested to properly align feature maps of multi-dimensional from different modalities. Experimental results on our private and public OpenfMRI datasets show that our proposed MFFormer performs better than that using a single modality or multi-modality MRI on schizophrenia and bipolar disorder classification.