PhysioGraph transformer a multimodal physiological emotion recognition personality aware dynamic graph transformer

The key to affective computing is emotion recognition based on multimodal physiological and behavioral responses and aids in intelligent systems to monitor mental health, adapt human-computer interaction, and predict the intent of affectivity. Although the existing deep models show remarkable advances, most of them process modalities in isolation and do not consider causal, temporal, and individual specific mechanisms underlying the expression of emotions. This paper presents PhysioGraph-Transformer which is a dynamic interaction-aware graph-based Transformer that encapsulates time-varying directed interactions among multimodal physiological channels with personality-sensitive adaptation. All the modalities of the AFFEC dataset, such as EEG, electrodermal activity (EDA), facial dynamics, eye gaze, pupil response, and cursor motion, are modeled as a time-varying interaction graph, which is learned with self-attention. A layer of Neural Ordinary Differential Equation (Neural-ODE) trains continuously changing latent embeddings to be able to smoothly model changes of emotion. Personality-conditioned cross-modes attention incorporates heterogeneous information into a common affective portrayal, making it give a meaningful and individual inference. Experimental analyses on the AFFEC indicate that PhysioGraph-Transformer achieves 84.6% accuracy and 80.8% macro-F1 which outperforms CNN, RNN, GCN and Transformer baselines and has strong resiliency to partial modality loss. The attention interaction maps illustrate distinguishable relationships between personality factors and modality inputs that offer neuroscientific information on the formation of emotions. These results make PhysioGraph-Transformer a interaction-aware, decipherable and personality conscious model of multi-modal physiological emotion recognition.