Multi-modal multi-task internet sentiment recognition model based on mid-to-back fusion strategy

Previous studies of multimodal sentiment recognition have paid limited attention to the discrepancy between unimodal expression and multimodal perception in sentiment labeling, and to the question of where in the network the modalities should be fused. This paper proposes a multimodal multi-task sentiment recognition model based on a mid-to-back fusion strategy. Text, audio and visual inputs are encoded with the pre-trained BERT, Wav2Vec 2.0 and CLIP models, respectively, and refined by convolutional, recurrent and self-attention layers. Each unimodal branch is trained with its own unimodal sentiment label, so that modality-specific representations are optimized before fusion; the three high-level unimodal representations are then integrated by a residual multi-head self-attention module supervised by the multimodal label, and the four tasks are optimized jointly. On the Chinese CH-SIMS dataset the proposed model achieves a three-class accuracy of 70.14% and a binary accuracy of 75.09%, improving on the strongest published multi-task baseline by 1.1 and 3.4 percentage points, respectively; the binary-accuracy gain is statistically significant, with p < 0.01 in Welch’s t-test. Fusion-position ablations show that delaying fusion raises neutral-class accuracy from 28.87% with early fusion to 43.82% with mid-to-back fusion, and the model transfers to the English CMU-MOSI benchmark, on which it reaches a binary accuracy of 86.53%. The model still has room for improvement with respect to dataset diversity, the complexity of real-world applications, non-standard language and cross-cultural sentiment understanding. Overall, the proposed approach improves the effectiveness of multimodal fusion and offers a new perspective for multimodal sentiment recognition.