Multimodal sentiment analysis based on expert feature-aligned diffusion under uncertain image modality

Abstract Multimodal sentiment analysis utilizes different types of information to improve sentiment recognition, driving research at the intersection of natural language processing and computer vision. However, practical applications often encounter missing image modal data due to collection restrictions, privacy protection, and other issues, which reduces the accuracy of sentiment analysis. Recently, diffusion models, as an emerging generative model, have demonstrated remarkable capabilities in handling modality completion and generation tasks. To this end, this paper presents an expert feature-aligned diffusion model (EFADM) integrated with curriculum learning, aiming to efficiently solve multimodal sentiment analysis tasks by dynamically generating missing modality data and optimizing feature alignment between modalities. EFADM consists of three primary modules: the Adaptive Noise Suppression Module (ANSM), the Expert Alignment Module (EAM), and a progressive generation framework based on curriculum learning. First, ANSM dynamically adjusts noise suppression during the diffusion process to enhance the quality of generated features. EAM strengthens semantic consistency between generated and existing modalities by aligning expert features. Additionally, the progressive generation strategy utilizing curriculum learning gradually increases task difficulty, thereby optimizing the model’s performance across different noise levels. Compared with existing methods, the proposed approach achieves leading performance on the publicly available MVSA series datasets.