Optimizing student engagement detection using facial and behavioral features

Abstract In recent years, computer vision and machine learning have achieved remarkable advancements across health care, autonomous systems, and robotics sectors. However, the educational domain, particularly the automated detection of student engagement in online and offline learning environments, remains rich in research opportunities. Detecting student engagement is inherently complex and requires sophisticated interpretive capabilities. This paper introduces a novel approach to automatically detect and classify student engagement by integrating facial images with behavioral and facial features, providing a comprehensive solution to enhance engagement recognition. This study begins by extracting behavioral features and correlating them with pre-defined engagement labels to perform engagement classification using machine learning techniques. In parallel, deep learning models are trained and validated on both image and behavioral features, offering a complementary approach. Additionally, the relationship between facial action units (AUs) and engagement labels is analyzed using three distinct metrics: conditional activation probability, relative activation ratio, and the statistical discriminant coefficient (SDC). The study utilizes publicly available datasets (WACV and DAiSEE) to perform extensive evaluations. Experimental results demonstrate that integrating action units significantly enhances model performance, with XGBoost achieving the highest accuracy (82.9%) among traditional models and EfficientNet reaching the best performance (47.2%) in deep learning experiments. These findings highlight the potential of multimodal approaches in improving real-time engagement detection, offering valuable insights into educational technologies and pedagogy.