Hand Raising Recognition in Remote Learning Environments Using Vision Transformer

Identifying when students raise their hands is a fundamental aspect of tracking engagement in remote learning environments. However, existing systems often face several key challenges, including scalability, robustness, interpretability of model outputs, and the accurate detection of subtle or partial hand-raising gestures. This paper proposes a framework using a Vision Transformer (ViT) to recognize hand-raising gestures from classroom images. The ViT captured spatial and contextual features from the image data. To evaluate the proposed ViT model, experiments were conducted on the Student Classroom Behavior (SCB) dataset, which includes activities featuring students’ hand-raising gestures and other behaviors. Gradient-Weighted Class Activation Mapping (Grad-Cam) was employed to provide visual explanations of what information the ViT model learned from images to recognize the hand-raising gesture. The proposed ViT model achieved a maximum average recognition accuracy of 93.82 ± 2.16% under 5-fold cross-validation, outperforming the baseline convolutional neural network model (CNN). The Grad-Cam outcomes show that the model focuses on valid features to determine its recognition. With the achieved level of performance, the proposed framework may assist in supporting immediate feedback loops, informing instructors of student queries, and promoting interactive remote learning experiences.