Enhancing Multimodal Sentiment Analysis with Hybrid Feature Representation and Deep Interaction

Multimodal sentiment analysis aims to infer user sentiment by integrating heterogeneous information from diverse sources, such as text and images. This study first conducts a comparative analysis of the capabilities of various pre-trained models for extracting sentimental features from both modalities. We then propose a method that enhances the representation power of unimodal features by explicitly fusing low-level sentimental features with deep semantic features. Specifically, for the visual modality, we construct a hybrid visual representation by combining deep semantic features from a pre-trained Convolutional Neural Network (CNN) with affective color and texture features. For the textual modality, we merge contextualized embeddings from a pre-trained language model with explicit sentiment scores computed from a lexicon, thereby injecting prior sentiment knowledge into the model. At the multimodal fusion level, we systematically compared various existing fusion strategies to effectively capture the modal interactions between visual and textual features. Experimental results on a publicly available image-text Twitter dataset demonstrate that our hybrid feature representation method yields significant performance improvements. Furthermore, the Transformer Encoder-based deep fusion strategy exhibits marked superiority over baseline methods, such as naive concatenation. This research validates that the synergy between enhanced hybrid feature representations and a deep interactive fusion strategy is an effective pathway for advancing multimodal sentiment analysis.