A novel multimodal personality prediction method based on pretrained models and graph relational transformer network

Multimodal personality analysis aims to identify and express human personality traits in videos. However, RNN and its variants have a limited ability to learn long-term temporal dependencies and existing methods neglect bimodal association features. Based on the fact that visual modalities play a dominant role in this task. Therefore, we propose a personality prediction method catering to learning and fusing intra-modal and intermodal feature dynamics. We first utilize pretrained models’ encoders to extract unimodal spatial scene features from videos. Then, we use xLSTM to capture sequence dependencies between different scene frames used as scene features. Meanwhile, we design a graph relational transformer network to learn longer intra-modal temporal interaction in three unimodal spatial features. Then, we calculate the similarity scores between visual and audio or text features as bimodal association features. Second, we design a multimodal attention feature fusion module to determine the contribution of each feature and aggregate these features. Finally, the MLP model is trained and used to predict scores for personality traits. Experiments on two benchmark datasets demonstrate that our method outperforms the existing methods and achieves state-of-the-art performance. Our code is available at https://github.com/RongquanWang/MP-PMGRT.