Markerless emotion recognition from full-body movements for Social XR

In this work, an emotion recognition system for enhancing social XR applications is presented. Although several techniques for emotion recognition have been proposed in the literature, they either require invasive and advanced equipment or exploit facial expressions, speech excerpts, physiological data, and text. In this contribution, on the contrary, an approach for markerless emotion classification through body language is designed. More specifically, human movements are analyzed over time by extracting the skeleton joints in videos acquired by consumer cameras. A normalization procedure has been introduced to provide a depth-independent skeleton representation without distorting the skeleton shape. The performance of the proposed method have been assessed using a dataset of videos recorded from multiple points of view. An ad-hoc learning-based emotion classifier has been trained to recognize four emotions (happiness, boredom, interest, and disgust) achieving an average accuracy of 72.5%. The pre-processed dataset, code, and demo with pre-trained models are available at https://github.com/michaelneri/emotion-recognition-human-movements . • We define a human skeleton representation that does not change with the distance between the user and the camera. • We introduce a new approach for classifying emotions based exclusively on body movements. • We extend an existing dataset employing multiple points of view for classifying four emotional states: happiness, interest, boredom, and disgust.