A novel inceptionTime framework for multimodal physiological signal-based mental state recognition using wearable device

Recognizing an individual's mental state from physiological signals is a topic of significant interest. However, due to the heterogeneity between different physiological signals, mental state classification models based on multimodal signals are still under development. In this study, we utilized a wearable device to collect electroencephalogram (EEG), electrocardiogram (ECG), and electromyogram (EMG) signals. We propose a novel convolutional neural network architecture that incorporates the InceptionTime module with multi-scale convolutional kernels and the SENET network to perform mental state classification based on multimodal signals. Specifically, we first designed an InceptionTime module to capture temporal features at different scales through convolutional kernels of different sizes. Then, the SENET network was added to the model to automatically focus on the key channels for mental state classification. Finally, a multimodal fusion module was designed to concatenate the features of all physiological signals. We evaluated the proposed model on self-collected emotion and fatigue datasets, selecting SVM, RF, EEGNET, and ResNET as baseline models. The experimental results demonstrate that modality fusion significantly enhances performance compared with a single modality, achieving the best mean accuracy of 78.06% for the recognition of three types of emotion (sadness, fear, and neutral) and a mean accuracy of 98.14% for fatigue detection. Moreover, compared to the optimal baseline method, our method respectively improved the accuracy by 14.88% and 10.99% on the emotion and fatigue datasets.