Schizophrenia Segmentation From sMRI Data: A Fusion Approach Using U-Net and Vision Transformer-Based Grad-CAM

Schizophrenia is a heterogeneous psychiatric condition with structural and functional abnormalities of the brain. The correct and meaningful detection of neuroanatomical features is essential for early diagnosis and treatment. In this work, we introduce a new fusion framework that combines Vision transformer (ViT) based Gradient-weighted class activation mapping (Grad-CAM) and U-Net model outputs for better feature extraction from structural magnetic resonance imaging (sMRI) scans. The sMRI data are spatially normalized with respect to a reference template based on healthy control subjects. Shannon entropy is used on sMRI to find the most informative slice with the most randomness. Three slices above and below the chosen slice are also taken for anatomical context, which gives us a 7-slice input per subject. These slices are then processed by a U-Net and a hybrid 2DCNN–Vision Transformer model. The ViT uses multi-head self-attention to learn long-range dependencies and produces Grad-CAM based attention maps to identify discriminative areas responsible for classification. On the other hand, a U-Net is trained solely on healthy subject slices to learn normative anatomical representations without supervision. During inference, schizophrenia slices are fed into the trained U-Net, and reconstruction errors are calculated. The absolute difference between the output and input images is thresholded to generate pseudosegmentation masks that highlight possible abnormalities such as ventricular dilatation, corpus callosum disturbance, and cortical abnormalities. By combining U-Net-derived anomaly maps with ViT attention maps, the model allows an interpretable and fine-grained localization of schizophrenia feature masks. Quantitative evaluation of the 2DCNN-ViT yields solid performance in all anatomical planes with up to 99.96% accuracy for coronal slices and 99.89% for axial and sagittal slices.