A multimodal feature learning framework with graph structure learning and CLIP for apparent personality analysis^*

Automatic personality prediction has become a challenging topic in computer vision. This paper proposes a novel multimodal feature learning framework with graph structure learning and CLIP for analyzing personality from short videos (GSL-CLIP). We construct a facial graph for the visual modality and propose a Geo method based on GCN and CNN to extract static facial appearance and geometric features. Furthermore, the pre-trained CLIP and VGGish CNN are used as feature extractors to extract their spatial features. Then, we design a graph relation attention network to mine their temporal feature in visual and audio modalities. Finally, we aggregate these two features through MLP to form a scene-audio association feature. Additionally, to enhance our framework’s robustness, we employ the pre-trained VGGish CNN for audio spatial features, and the BiRNN with an attention mechanism is used to extract significant audio temporal features. Finally, we introduce a multimodal channel residual attention network for fusing multimodal features and leverage an MLP regression model to predict scores for personality traits. The experimental results indicate that the effectiveness of our proposed GSL-CLIP is superior to state-of-the-art methods.