A Siam-LSTM Model for Multichannel EEG on VR Motion Sickness Recognition

The combination of prior knowledge of electroencephalography (EEG) and deep learning always receives better results in EEG pattern recognition. Thus, we propose end-to-end models combining functional brain network (FBN) and Siamese long short-term memory (Siam-LSTM) model and apply them to a virtual reality motion sickness (VRMS) recognition task. Siam-LSTM is a key module in the proposed models to process EEG signal electrode by electrode via the shared layers. The models apply 1D-convolutional neural network (CNN) and Siam-LSTM to extract temporal features, and then, Pearson correlation coefficients of the features among the electrodes are computed as the FBNs. The 2D-CNN and dense networks are used to extract the FBN feature and do regression analysis. In final, a VRMS-related EEG dataset is used to verify the models by tenfold cross-validation, while the VRMS level is assessed by the subjective simulator sickness questionnaire (SSQ). The experimental results show that the two major proposed models named BC-CNN-LSTM and BC-CNN-LSTM2 obtain the best performance in regression, i.e., mean square errors (mses) between the predicted SSQ scores and the real SSQ scores are <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$26.89 \; \pm \; 8.89$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$24.75 \; \pm \; 4.70$ </tex-math></inline-formula>, respectively. The proposed models provide a new idea for long short-term memory (LSTM) to process multielectrode EEG signals and outperform the state-of-the-art methods in VRMS recognition.