Deep-MultiAPP: First-impression based apparent personality trait prediction from short videos with multimodal deep learning

Abstract Automated Apparent Personality Prediction (APP) from multimodal data holds significant promise for enhancing Human-Computer Interactions (HCIs), yet existing models often fall short in effectively fusing diverse cues like visual expressions, audio prosody, and textual semantics, leading to suboptimal accuracy and interpretability in real-world applications. This study introduces Deep-MultiAPP, a novel multimodal deep learning architecture designed to address these limitations by integrating Residual Network (ResNet-50) and Vision Transformer (ViT-B/16) for hybrid visual feature extraction, Gated Recurrent Unit (GRU) for temporal modeling, enhanced audio Multilayer Perceptrons (MLPs) for incorporating mean-pooled log-Mel spectrograms, and Bidirectional Encoder Representations from Transformers (BERT) for text embeddings with late decision-level fusion combines modality-specific regressors (weights: vision 0.6, audio 0.3, text 0.1). On the ChaLearn First Impressions V2 dataset, Deep-MultiAPP achieved a state-of-the-art mean regression accuracy of 91.77% across the Big-Five traits, outperforming prior baselines by ∼0.25–0.47%. Binarized classification at a 0.5 threshold yielded a macro-averaged precision of 79.54%, recall of 80.13%, and F1-score of 79.83%, as validated by confusion matrices showing balanced error distributions despite class imbalances. These results demonstrate Deep-MultiAPP's superior capability in capturing nuanced personality cues, paving the way for more reliable artificial intelligence (AI) systems in fields like recruitment, mental health screening, and social robotics, while highlighting the need for bias audits and cross-cultural validations to ensure equitable deployment for responsible real-world use.