Emotion Recognition in Interactive VR Tasks and in 360VR Videos with Transformer-Based Approach and Multimodal Sensing

We are comparing emotion recognition models trained on passive and active emotional stimuli. Interactive Virtual Reality (VR) systems have been shown to elicit strong emotions, but Artificial Intelligence (AI) emotion recognition classifiers have primarily been trained on physiological data elicited by passive stimuli (e.g. videos), with less consideration for interactive VR tasks. We present a study designed to elicit emotions through both passive (VR videos) and active (interactive VR tasks) stimuli. From our study, we designed a dataset with multiple input modalities, including electroencephalography (EEG), electrodermal activity (EDA), heart rate (HR), and eye-pupil response. To analyze our multimodal dataset, we designed an emotion recognition classifier using deep learning-based transformer and fully convolutional network architectures. Our evaluation shows that the model trained on interactive VR tasks achieved best accuracy around 97% with a significant role of EEG in multimodal emotion recognition.