EEG and Eye-Tracking Signal Fusion for Mild Depression Identification Under Visual Illusion Stimulation

To overcome the subjectivity inherent in traditional depression diagnosis, this study explored the potential for objective detection using multimodal physiological signals and deep learning. Electroencephalogram (EEG) and eye-tracking data were collected from healthy controls (HC) and a mild depression (MD) group while they viewed a rotating tunnel visual illusion stimulus. Functional connectivity was characterized using Pearson correlation coefficient matrices derived from EEG signals, and eye-tracking trajectory data were processed. A dual-stream deep learning model based on feature-level fusion was proposed; the EEG pathway employed a CNN-Transformer architecture to extract spatio-temporal features, while the eye-tracking pathway utilized a Transformer model to capture temporal dependencies. Evaluated using ten-fold cross-validation, the results demonstrated that the feature-level fusion multimodal model exhibited the best performance in distinguishing between HC and MD groups, achieving an accuracy of 88.30%, precision of 90.27%, recall of 88.43%, and an Fl-score of 87.52<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">%</sup>. This outperformed the best unimodal models with EEG accuracy: 85.00%, eye-tracking accuracy: 78.33 %, and a decision-level fusion model with accuracy: 85.00%. This research confirms that fusing EEG functional connectivity and eye-tracking features can effectively enhance the performance of objective detection for mild depression.