A Gesture-centered Dual-modal Network for Micro-Gesture Emotion Recognition

To address the issues of insufficient modality collaboration, difficulty in modeling fine-grained dynamic features, and underutilization of gesture information in current micro-behavior emotion recognition, we propose a Gesture-centered Dual-modal Network for micro-gesture emotion recognition(GDN) based on RGB-Heatmap two-stream fusion. First, the framework employs a dual-modal feature interaction mechanism to achieve deep complementarity and cooperative enhancement between RGB visual information and skeleton gestural features. Second, we introduce a Res2Net3D-based three-dimensional multi-scale feature extraction network that integrates 3D convolutions with a multiscale residual architecture, effectively enhancing the perception of spatiotemporal dynamic information across different scales. Additionally, a self-adaptive gesture attention module is designed to improve the modeling of emotional state variations within the heatmap modality. Finally, the proposed method is evaluated on the iMiGUE dataset and compared with algorithms such as TSM+LSTM and PoseC3D, further validating its effectiveness and robustness.