Micro-expression recognition based on the fusion of facial landmark points and optical flow features

Micro-expressions are fast and subtle spontaneous facial movements with extremely short reaction times, and due to their secretive nature, micro-expressions are often not easy to detect, but they can truly reveal people’s inner emotional world. As the low-intensity and small-scale datasets of micro-expressions provide very limited information and the input of a single feature cannot extract more useful information, we propose a dual-stream 3D micro-expression recognition algorithm that combines face landmark features and optical flow features. By introducing the face landmark branch, we can capture the motion changes of the key parts of the face more accurately, which makes up for the limitations of the optical flow method under light changes. In addition, the softmax loss in the traditional classification task is changed to the Am-softmax loss function, which can reduce the intra-class distance to increase the inter-class distance. Our method has been extensively experimented on three publicly available datasets (i.e., Spontaneous micro-expression corpus, Consortium for Advanced Micro-Expression Research, and Spontaneous Actions and Micro-Movements) using leave-one-out cross-validation, and the experimental results show that the proposed method achieves excellent performance in micro-expression multiclassification and outperforms the state-of-the-art five-class expression classification methods.