IBFNet: A Dual Auxiliary Branch Network for Multimodal Hidden Emotion Recognition

Micro-expression (ME) is a crucial cue to reveal hidden emotions and helps to diagnose mental illnesses such as depression. However, their rapid and subtle characteristics make them difficult to recognize, and relying on a single ME feature is insufficient to capture comprehensive information about hidden emotions. This paper proposes an inverted bottleneck fusion network (IBFNet), which combines ME and facial tissue oxygen saturation (StO2) for multimodal feature fusion to improve the recognition of hidden emotions. Specifically, IBFNet learns different representations of each modal feature, namely common and individual features. The modalities are then mapped to different subspaces for feature refinement. We design the recurrent cross-modal attention (RCA) module to explore commonalities across modalities, retaining modality-specific information through auxiliary branches to enhance diversity. Finally, the inverted bottleneck structure is used to fuse the common and individual feature of ME and StO2. Experimental results demonstrate a recognition accuracy of 90.47%, which surpasses the limitations of single-modality recognition.