CHTMAE: Cross-Modal Hierarchical Temporal-Spatial Masked Autoencoder Model for Micro-Expression Recognition

Most Micro-expression recognition (MER) methods tend to explore only single modality (key frames or optical flow), ignoring intermediate frames, which usually results in the omission of critical temporal cues. Meanwhile, the generalization of MER deep models remains limited due to insufficient annotated micro-expression (ME) samples. To address these challenges, this paper proposes an efficient self-supervised MER scheme based on deep learning, the cross-modal hierarchical temporal-spatial masked autoencoder framework, named CHTMAE. Specifically, CHTMAE constructs a reconstruction task via both optical flow- and image-based masked autoencoders (MAE) for the multi-feature representation on motion and spatial features from unlabeled ME data. To obtain spatiotemporal fusion related to the action unit (AU) in the priors of MEs, this work designs the Hierarchical Temporal-Spatial Feature Guided Enhancement Component (H-TSGEC). Generally, the H-TSGEC consists of a cascade of Temporal-Spatial Feature Guided Enhancement Units (TSGEUs) which further comprise the Spatiotemporal Guided Cross-Attention Module (SGCAM) and the Frame Token Attention Module (FTAM). By cascading multiple TSGEUs, the CHTMAE achieves progressive fusion for diverse motion representation on subtle MEs. Extensive experimental results validate the effectiveness of the CHEMAE model, whose performance surpasses that of state-of-the-art MER methods. The source codes for this model are available on: https://github.com/cc99forever/CHTMAE99/tree/master.