Improved human emotion recognition from body and hand pose landmarks on the GEMEP dataset using machine learning

Emotion recognition has gained popularity in recent years. However, it is still a very challenging task due to the complexity, variety and multi-modality of human emotion expression. Focusing on visual data, body language plays an important role in emotion recognition, although this topic remains under-explored. This paper presents a thorough study of emotion recognition using seven machine learning techniques applied to three different configurations of human skeletal landmarks. A first configuration uses a reduced set of body pose landmarks obtained by MediaPipe. A second configuration also uses MediaPipe, but focuses on obtaining hand pose landmarks. Finally, the third configuration selects a number of landmarks from the previous two configurations. Five basic emotions ( joy , pride , anger , sadness and fear ) from the multimodal GEMEP dataset are considered for comparison. Remarkably, the best accuracy for all configurations was greater than 99.5%. The highest accuracy was achieved by Random Forest (RF) classifier for the first two configurations (99.82% and 96.43% respectively), closely followed by Gradient Boosting (GB). The third configuration showed an accuracy of 98.81% by GB, followed by 97.62% by RF. It should be noted that our proposal achieves an improvement in accuracy of almost 4% compared to other approaches using the GEMEP dataset. • Real-time emotion recognition based on skeletal landmarks of body & hands is presented. • Seven ML techniques are applied to 3 different configurations of human body landmarks. • Configurations use body pose landmarks, hand pose landmarks and a mixture of both. • The comparative study shows accuracy values greater than 99.5% of emotion recognition. • Random Forest and Gradient Boosting outperform all other ML techniques used.