Open Vocabulary Emotion Prediction Based on Large Multimodal Models

The Multimodal Emotion Recognition (MER 2024) Challenge focuses on recognizing emotions through the integration of audio, language, and visual signals, driving advancements in the field of affective computing. This study presents our approach for the MER-OV sub-challenge, concentrating on open-vocabulary emotion recognition. We innovatively employ Optical Character Recognition (OCR) technology to optimize video subtitles, thereby enhancing the accuracy of textual descriptions. Additionally, we utilize in-context learning techniques with large language models (LLMs) for open-vocabulary emotion prediction. By incorporating video content analysis, we leverage large multimodal models (LMMs) to further improve the accuracy of emotion prediction. Our proposed text-only modality-based open-vocabulary emotion prediction method achieves an average score of 51.0% on the training set, and the multimodal open-vocabulary emotion prediction method achieves an average score of 59.1%. This surpasses the best model in the baseline, GPT-4V, which has a score of 56.0%, achieving a state-of-the-art (SOTA) result.