Multimodal Dynamic Face Emotion Verification Model based on Priori Prompt Words for Dynamic Facial Expression Recognition

The recognition and classification of facial emotions are crucial in the field of education. With the rapid advancement of artificial intelligence, the innovative application of deep learning for Dynamic Facial Expression Recognition (DFER) in videos has become essential for evaluating educational outcomes. While multimodal models have shown great promise in DFER scenarios, the fusion of language modalities describing facial expressions remains underexplored. Furthermore, for video data, focusing on the temporal relationships between frames is critical. We propose the Multimodal Dynamic Face Emotion Verification Model based on Priori Prompt Words (MPPW-DFER) to address these challenges. Our model utilizes a large multimodal model to generate Priori Prompt Word (PPW) descriptions from video data, which are then used as input for multimodal fusion. For dynamic videos, we introduce the Temporal Deformable Attention Embedding (TDAE), which enhances the temporal attention mechanism during the initial input phase. Our approach outperforms existing methods on two public datasets, DFER and MAFW, demonstrating its effectiveness and innovation in the field.