Micro-expression Spotting based on Multi-modal Hierarchical Semantic-guided Deep Fusion Model*

Most existing micro-expression spotting (MES) methods predominantly learn from optical flow features while neglecting the detailed information contained in RGB images and few MES works have explored the optimal interaction fusion of the optical flow modality and RGB modality. To address these issues, we propose a multi-scale layered semantic-guided end-to-end cross-modal fusion framework for MES with a convolutional neural network (CNN)-Transformer, named MESFusion. To obtain cross-modal complementary information, this scheme sequentially constructs a Multi-Scale Feature Extraction Module (MFEM) and a Hierarchical Cross-modal Semantic Guidance Fusion Module (H-CSGFM). Guided by the multi-scale fused features extracted by the MFEM, the Spatial Cross-Attention Fusion Unit (SCAFU) is designed to achieve comprehensive fusion and global interaction of complementary information between optical flow and RGB images, while maintaining the intrinsic characteristics of each modality during the fusion of different modalities. The Channel Self-Attention Fusion Unit (CSAFU) is introduced to model the correlation between non-scale fused features and source image features in the channel dimension and to filter out interfering features from the source images. Furthermore, by hierarchically integrating SCAFU and CSAFU, we capture diverse informative facial cues. Extensive experiments conducted on the CAS(ME<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">)2</sup> dataset and the SAMM Long Videos dataset demonstrate that our MESFusion model surpasses competitive baselines and achieves new state-of-the-art results.