Cultural differences in facial features based on SHAP analysis from machine learning model estimating learners' mental states

Estimating the mental states is important for tracking students’ e-learning performance. A previous study by our team revealed that students’ facial expressions can estimate Japanese learners’ engagement and help-seeking states. However, whether a model made for a single culture can generalize to other cultures is unknown. The current study applied the prediction method for the Japanese to the Taiwanese and compared the results with the two cultures. Participants solved a linguistic problem on an intelligent tutoring system (ITS), and their facial videos, clicks of hint buttons, and answers were recorded. Facial features, the Action Units (AUs), were used for the LightGBM (Light Gradient Boosting Machine), a machine learning method, to classify the mental states. The Shapley Additive exPlanations (SHAP) analysis was used to explain feature contributions. For estimating the engagement state, the top ten important features were more around the eyes (e.g., AU02) than those around the mouth (e.g., AU23) for Japanese, while that was the opposite for Taiwanese. In contrast, for help-seeking, more common elements between the two cultures were in the top ten important features, such as AU04 (brow lowerer) and AU20 (lip stretcher). Furthermore, the paired t-test results showed a significant difference in the SHAP values for estimating the help-seeking states between Taiwan and Japan but not for the engagement states. The discussion focused on the sum of the SHAP values. The current study revealed cultural effects in the relationship between mental states and facial features based on SHAP analysis.