SCL-GT: Supervised Contrastive Learning with Graph Neural Networks and Transformer for Upper-Body Gesture Emotion Recognition

Inertial sensing technology brings new possibilities for emotion recognition by addressing occlusion and spatial constraints in visionbased solutions. Nevertheless, fully exploiting the interactive potential of inertial sensors remains a challenge. In this paper, we propose a supervised contrastive learning with graph neural networks (GNNs) and Transformer (SCL-GT) architecture to recognize emotions from upper-body gestures using inertial sensors. Specifically, SCL-GT leverages GNN and Transformer branches to extract complementary emotional features from motion data in the local structure view and global interaction view, respectively, and introduces a supervised contrastive learning task to ensure consistency across cross-view representations. For the GNN branch, a graph representation strategy is developed to model the relationships among sensors, and a dynamic adjacency matrix is integrated into graph convolutional networks (GCNs) to improve upper-body gesture emotion recognition. Furthermore, we construct the EmoUP dataset, which captures eight emotions from the upper-body gestures of sixteen participants using five inertial sensors. Experimental results show that SCL-GT outperforms state-of-the-art methods in terms of overall performance across both positive and negative valences on the EmoUP dataset.