Predicting learners' engagement and help-seeking behaviors in an e-learning environment by using facial and head pose features

In an e-learning environment, it is difficult for teachers to track learners’ engagement or detect when they need help. The current study estimates two mental states: the engagement state and the help-seeking state. We asked participants to solve a problem on an intelligent tutoring system (ITS) and recorded their facial videos, clicks of hint buttons, and answers. Action Units (AUs) and head pose features were extracted from OpenFace to consist of three feature sets: Basic AUs, Head Pose, and Co-occurring AUs feature sets. LightGBM (Light Gradient Boosting Machine) and SVM (support vector machine) classifiers showed 0.69 to 0.93 accuracy in estimating the two mental states. The classification performance revealed that LightGBM is better than SVM. We used SHAP (Shapley Additive exPlanations) analysis to evaluate the importance of Basic AUs and Head Pose features. The results showed that AU02 (outer brow raiser), AU23 (lip tightener), and AU04 (brow lowerer) are important for estimating the engagement states; AU04, AU23, and AU14 (dimpler) are important for estimating the help-seeking states. The current study succeeded in estimating when participants are engaging in solving a problem and when they need help. Features obtained from facial videos are useful in improving e-learning education. • We used facial features and head pose features to estimate engagement and help-seeking states separately. • We used a machine learning method, LightGBM. • Both facial features defined as Action Units and head pose features are useful. • The accuracy of predictions is 0.79 for engagement and 0.93 for help-seeking states. • SHAP analysis suggested that the important features are different around the mouth, eyes, and head position.