MER-MFNet: Micro-Expression Recognition Based on Multimodal Fusion Network

Micro-expressions (MEs) are brief and subtle facial movements that can reveal hidden emotions. We observe that MEs are manifestations of small localized changes, and deep learning has already played an important role in this area. However, existing studies do not adequately consider the local and global characteristics of the data: apex frames represent global apex features, while optical flow maps reflect localized subtle movements. Therefore, different data modalities require the use of suitable feature extraction network architectures. In this paper, we propose a novel multimodal fusion network for micro-expression recognition (MER-MFNet). The network comprises three components: apex frames feature learning using vision transformers (ViT), optical flow maps feature selection using convolution neural networks (CNNs), and micro-expression (ME) keypoint feature classification. The apex frames and optical flow maps features are fused using a convex combination method, and their similarity is compared. Finally, the combined features are integrated with the keypoint classification results to achieve efficient MER. Experiments on CASME II, SAMM, SMIC-HS and MEGC2019-CD datasets show that MER-MFNet exhibits excellent performance in MER, significantly improving recognition accuracy and robustness.