Emomamba: Advancing Dynamic Facial Expression Recognition with Visual and Textual Fusion

This paper presents EmoMamba, a novel framework designed for dynamic facial emotion recognition by using both visual and textual input. The proposed approach consists of a feature encoder stacked with our proposed GBMamba layers. Initially, a vision backbone is trained on a static facial emotion dataset to extract meaningful features. Subsequently, cropped facial frames are encoded using the vision backbone to produce sequences of facial features. These sequences are processed by an encoder that incorporates GBMamba layers. Moreover, the video description text is encoded using the Mamba module. The visual and textual features are then concatenated and passed through an EmoMamba multimodal encoder to effectively integrate both modalities. Experimental evaluations on dynamic facial emotion datasets, such as MAFW and DFEW, demonstrate the effectiveness and potential of the proposed method.