T-SMCFN: An EEG-based Framework for Macro-expression, Micro-expression, and No-expression Recognition

Expression recognition based on computer vision methods has been remarkably successful. Still, it is not as effective when the facial image is affected by factors such as lighting, pose, and occlusion. Electroencephalography (EEG) signals have very high temporal resolution, which can accurately capture real changes in brain activity and achieve good recognition results in many scenarios. In this study, an independent recognition task based on EEG signals for macro-expressions, micro-expressions, and no-expressions was carried out. First, we collected facial images and EEG data of macro-expressions, micro-expressions, and no-expressions from 200 subjects. We constructed SWUE, a three-category expression dataset containing six emotions (happy, disgust, sadness, anger, fear, and surprise). Second, the Temporal Spatial Multi-scale Convolutional Fusion Network (T-SMCFN) is proposed for fusing temporal and spatial features for three types of expression classification recognition. We experimented with data from a single emotion and a mix of six emotions, and grief obtained the highest average classification accuracy of 98.44% for the single emotion data and 94.90% for the mix of six emotions data. The results show that the proposed T-SMCFN architecture has excellent stability and also provides a tractable method for macro-expression, micro-expression, and no-expression recognition studies based on EEG signals.