Fine-Tuning Methods of Multimodal Pretraining Models for Emotion Recognition

With the continuous progress of artificial intelligence, single-modal models are increasingly falling short of meeting the requirements of complex tasks. As a result, there has been a burgeoning interest in multimodal pretraining models. Traditional single-modal models, like those in natural language processing that concentrate solely on text or computer vision models that focus only on images, have limitations when dealing with multimodal data. They are unable to fully exploit the complementary and associative information among different modalities. The development of multimodal models has been driven by significant growth in computational power, the availability of large-scale multimodal datasets, and breakthroughs in deep learning technology. These advancements empower models to learn feature representations from diverse modalities and establish cross-modal associations and fusion. This paper centers on the current research scenario of multimodal pretraining models in the field of emotion recognition. It analyzes mainstream model architectures and methodologies, as well as their efficacy in identifying various emotional expressions. Through a comparison of existing fine-tuning techniques, this study explores strategies for attaining optimal performance in multimodal emotion recognition. Additionally, this paper discusses the principal challenges that current multimodal pretraining models face in emotion recognition, such as data scarcity and model interpretability. It also contemplates potential future research directions. This study aims to provide a comprehensive reference for researchers in the field, thereby facilitating further advancements and applications of multimodal emotion recognition.