Multimodal emotion recognition method in complex dynamic scenes

Multimodal emotion recognition technology leverages the power of deep learning to address advanced visual and emotional tasks. While generic deep networks can handle simple emotion recognition tasks, their generalization capability in complex and noisy environments, such as multi-scene outdoor settings, remains limited. To overcome these challenges, this paper proposes a novel multimodal emotion recognition framework. First, we develop a robust network architecture based on the T5-small model, designed for dynamic-static fusion in complex scenarios, effectively mitigating the impact of noise. Second, we introduce a dynamic-static cross fusion network (D-SCFN) to enhance the integration and extraction of dynamic and static information, embedding it seamlessly within the T5 framework. Finally, we design and evaluate three distinct multi-task analysis frameworks to explore dependencies among tasks. The experimental results demonstrate that our model significantly outperforms other existing models, showcasing exceptional stability and remarkable adaptability to complex and dynamic scenarios. • Designs a robust multimodal emotion recognition method with dynamic-static feature fusion to handle complex scenes. • Leverages stacked LSTM and contrastive learning for enhanced emotional feature extraction and representation. Introduces D-SCFN for efficient multimodal feature fusion within the T5 model architecture. • Employs multi-task frameworks to optimize dependencies among emotion recognition tasks. Fading and monitoring by a peer combined foster domain-general strategy knowledge.