PSACF: Parallel-Serial Attention-Based Cross-Fusion for Multimodal Emotion Recognition

Multimodal emotion recognition (MER) has significantly improved by integrating features from various modalities. However, imbalances and heterogeneity among modalities often result in information redundancy and distributional differences. To this end, we propose a Parallel-Serial Attention-based Cross-Fusion (PSACF) framework that includes feature preprocessing, feature extraction, and multimodal cross-fusion. Initially, we align modalities in both temporal and feature dimensions. Next, we employ a collaborative encoder to capture modality-invariant information, while a sole encoder extracts modality-specific data. Then, we introduce a multimodal cross-fusion module with collaborative (Col) and collaborative guide sole (CSol) tasks to enhance inter-modal correlations and address imbalances. Col uses self-attention for long-distance dependencies, and CSol optimizes fusion via parallel-serial attention. Additionally, we integrate multiple loss functions for joint optimization to mitigate intermodal heterogeneity. Extensive experiments on the CMU-MOSI and CMU-MOSEI datasets demonstrate that PSACF achieves competitive performance, surpassing state-of-the-art methods.