Emotion recognition: Low-rank Multimodal Shear and Splicing Fusion

We propose a low-rank multi-modal shearing and splicing fusion (LMSSF) method for accurate emotion recognition by effectively integrating the information from three modalities: text, image, and voice. Recognizing users’ emotions accurately using social media information is challenging due to the diversity of user-generated data and the difficulty in accurately identifying and extracting features from multi-modal information. Our method fills the gap in the multi-modal field by leveraging feature extraction and fusion techniques to combine voice, image, and text modalities for emotion recognition. To address the interaction of multi-modal information, we introduce standard feature extraction and private feature retention methods to ensure the integrity of the multimodal information. Further-more, we have developed a step-by-step discrimination approach that significantly reduces calculation and discrimination time by distinguishing standard features of three modalities, common features of two modalities, and private features. Our method effectively solves the problem of accurately recognizing emotions in social media users with diverse information modalities, achieving 97.3% accuracy with only 180K and an accuracy improvement of close to 10% than others. Code is available at https://github.com/wjm666666/LMSSF.