A multimodal personality prediction framework based on adaptive graph transformer network and multi‐task learning

Abstract Multimodal personality analysis targets accurately detecting personality traits by incorporating related multimodal information. However, existing methods focus on unimodal features while overlooking the bimodal association features crucial for this interdisciplinary task. Therefore, we propose a multimodal personality prediction framework based on an adaptive graph transformer network and multi‐task learning. Firstly, we utilize pre‐trained models to learn specific representations from different modalities. Here, we employ pre‐trained multimodal models' encoders as the backbones of the modality‐specific extraction methods to mine unimodal features. Specifically, we introduce a novel adaptive graph transformer network to mine personality‐related bimodal association features. This network effectively learns higher‐order temporal dependencies based on relational graphs and emphasizes more significant features. Furthermore, we utilize a multimodal channel attention residual fusion module to obtain the fused features, and we propose a multimodal and unimodal joint learning regression head to learn and predict scores for personality traits. We design a multi‐task loss function to enhance the robustness and accuracy of personality prediction. Experimental results on the two benchmark datasets demonstrate the effectiveness of our framework, which outperforms the state‐of‐the‐art methods. The code is available at https://github.com/RongquanWang/PPF-AGTNMTL .