Diagnosis of Schizophrenia Using Multimodal Data and Classification Using the EEGNet Framework

<b>Background/Objectives</b>: In recent years, people have been facing a lot of difficulties in handling stress, emotions, social, and behavioral issues, which have led to severe mental disorders. Schizophrenia is one disorder that requires more attention. This disorder is characterized by positive or psychotic symptoms, negative symptoms, and cognitive symptoms, which makes diagnosis and treatment complicated. The main objective is to identify the degree of severity of symptoms through multimodal data and classify them using the EEGNet framework. <b>Methods</b>: Multimodal data are collected. To identify the severity of symptoms of schizophrenia, initial screening is performed through assessment tools such as the Positive and Negative Symptoms Scale (PANSS), Brief Negative Symptom Scale (BNSS), Scale for Assessment of Negative Symptoms (NSA-16), and Scale for Assessment of Positive Symptoms (SAPS). Designed photo elicitation and VR box video stimuli are used for data collection. The patients are asked to express their thoughts upon viewing photos shown through a photo elicitation task. The patients are given Virtual Reality (VR) stimuli where videos will be played in a VR box and patients are asked to express their thoughts. Patients' facial expressions and speech signals are captured through webcam while performing these tasks. Finally, the electrical activities of the patients are assessed through a 14-channel EEG headset. A novel method of fusing and embedding of normalized multimodality features into the EEGNet architecture is carried out that enables combined utilization of electrophysiological information from EEG and complementary behavioral-affective cues from other modalities, thereby enhancing classification performance while retaining the architectural efficiency of EEGNet. <b>Results:</b> The reliability and validity of the questionnaire are statistically analyzed and found to be α = 0.761. The sum of variance of PANSS is about 27.08, SAPS is about 28.61, and BNSS is about 29.92 with <i>p</i> < 0.05. This EEGNet model displays an accuracy of 0.99, precision of 0.99, recall of 0.98, and F1-score of 0.99 for healthy and a precision of 0.98, recall of 0.99, and F1-score of 0.99 for schizophrenia-affected patients and ROC AUC of about 0.9989. <b>Conclusions</b>: This system proves to be a promising method for the diagnosis of schizophrenia and thereby enhances the performance of the system.