Perception and Structure Association Network for Micro-Expression Recognition

Micro-expression recognition (MER) poses a formidable challenge due to the fleeting and subtle nature of micro-expressions (MEs). Current approaches predominantly focus on either motion features or facial structure, yet fail to effectively integrate both perspectives. This paper introduces a novel Perception and Structure Association Network (PSA-Net) for MER, aiming to effectively extract key features in MEs by performing multi-scale perception of motion features and constructing adaptive facial structure association. Specifically, to perceive fine-grained ME features and achieve multi-scale key feature extraction, we have designed the Multi-scale Micro-motion Perception module. This module extracts motion features by dividing windows and performing self-attention calculations within each window. The motion features of each window are then fused layer by layer using shifted window attention and spatial attention. Simultaneously, to effectively construct the structure association of facial ME regions, we have designed the Adaptive Facial Structure Association module. This module employs graph attention mechanisms and global adaptive attention maps to achieve the adaptive structure association of facial regions of interest, enabling the constructed facial graph structure to effectively extract global ME information and the relationships between nodes. Additionally, to effectively fuse these two key ME features, we propose the Fusion Loss, which can effectively learn the similarities between them, promoting the model to learn more discriminative feature representations, thus achieving effective feature fusion. Extensive experiments conducted on four public ME datasets (CASME, CASME II, SAMM, CAS(ME)3) have achieved recognition accuracy/F1 scores of 81.87%/79.26%, 87.80%/88.08%, 82.35%/76.76%, and 63.10%/49.47% respectively, surpassing most existing methods, which indicates that our method can effectively improve MER performance.