Monitoring of Student Classroom Learning Status based on Hybrid ViT-YOLOv5s

In modern era, the monitoring of classroom learning status plays a crucial role in educational technology, which enables trainers or instructors to evaluate students’ engagement and performance, by improving teaching strategies. However, the Convolutional Neural Network (CNN) based approach, failed in capturing long range dependencies among the data and struggles with occlusions and small object detection. To overcome limitations, a novel based approach, hybrid Vision Transformer-YOLOv5s (ViT-YOLOv5s) is proposed for enhanced students’ behavior and engagement recognition. Initially, the data is collected from the AffectNet and classroom action datasets for detection of emotions and activity. After that, the collected images are normalized using min max normalization to handle the occlusions. Furthermore, the ViT is utilized to extract local and global features. Finally, the YOLOv5s is used to perform behavior recognition of students in classroom and DeepSORT algorithm is employed for tracking and engagement monitoring of students. The proposed hybrid ViT-YOLOv5s attained better results in precision of 0.80, recall of 0.79, mean Average Precision ([email protected]) of 0.82 and f1-score of 0.79, and outperformed existing YOLOv5s model.