SEPT: Uncovering Student Difficulties through Emotional and Physical Trajectories during Online Assessments

Imagine a classroom where every student’s struggles and triumphs are revealed not through words but via the silent language of their faces and movements captured in real time as they tackle a quiz. Modern higher education is increasingly moving towards blended and remote learning, providing flexibility but reducing face-to-face interaction. Without direct visual cues, instructors struggle to promptly detect when students are having difficulties. To address this challenge, we introduce the concept of Student Emotional and Physical Trajectories (SEPT). We captured, via laptop webcams, and analysed subtle facial expressions and behaviours of 89 first year Master’s students during machine learning quizzes. We extracted emotional features (valence and arousal on Russell’s circumplex model) and physical states (head orientation and distance to the screen) on a second-by-second basis using a computer vision pipeline including face detection with YOLOv5 and valence and arousal deep predictors networks. The result is a seven-dimensional time series for each student’s quiz attempt. Using Dynamic Time Warping (DTW) and clustering, we identified distinctive patterns in these trajectories that differentiate between difficulties arising from individual student factors and those related to specific quiz questions. Our results show that moment-to-moment fluctuations in emotional and physical states provide valuable insight into student behavior and performance. This work bridges affective computing and education, paving the way for intelligent systems that monitor student affect and behavior in technology-driven classrooms.