SZAtt-Net: A unified deep learning model with different attention mechanisms for schizophrenia classification from multimodal data

Schizophrenia (SZ), a complicated mental illness, shows up as incorrect beliefs, perceptual distortions, poor reasoning, and neurocognitive problems. Its varied character—varying in severity, development, and therapy response—offers major diagnostic difficulty. While Magnetic Resonance Imaging (MRI) provides comprehensive structural and functional imaging for a better knowledge of the condition, Electroencephalography (EEG) with its great temporal resolution offers vital insights into neuronal malfunction in SZ. Deep learning (DL) techniques as well as advanced machine learning (ML) have been developed to improve SZ detection, therefore facilitating quick identification and improved patient recovery. This study presents SZAtt-Net, a DL framework designed for SZ detection and classification, integrating Convolutional Neural Networks (CNNs), Bidirectional Gated Recurrent Unit (BiGRU), along with Multilayer Perceptron (MLP) architectures. A key contribution of this work is the comprehensive ablation study of channel, self, spatial, and temporal attention mechanisms in DL, conducted to assess their impact on model performance across multimodal data. Notably, due to the absence of dedicated MRI datasets for SZ classification, this study repurposes an MRI segmentation dataset for classification, making it the first such attempt. As a unified model, SZAtt-Net is applied separately to three benchmark datasets—Kaggle EEG, LMSU EEG, and Hippocampus MRI—achieving accuracy rates of 99.37% and 98.92% using channel attention, and 96.33% using spatial attention, respectively. The proposed framework is also benchmarked against various pre-trained models, and a Gradient-weighted Class Activation Mapping (Grad-CAM) analysis is performed to enhance interpretability. This work underscores the clinical relevance of SZAtt-Net and outlines future research directions for improving accessible and accurate diagnostic solutions for SZ. • Proposed SZAtt-Net model for multimodal detection and classification of Schizophrenia. • Integration of multimodal imaging data from both MRI and EEG. • Proposed framework tested on 3 open-access multimodal Schizophrenia datasets. • Our framework’s decisions have been explained via Grad-CAM analysis. • Comparison of the SZAtt-Net model with other state-of the-art models.