Deep-Learning-Based Human Activity Recognition: Eye-Tracking and Video Data for Mental Fatigue Assessment

This study addresses mental fatigue as a critical state arising from prolonged human activity and positions its detection as a valuable task within the broader scope of human activity recognition using deep learning. This work compares two models for mental fatigue detection: a model that uses eye-tracking data for fatigue predictions and a vision-based model that relies on vital signs and human activity indicators from facial video using deep learning and computer vision techniques. The eye-tracking model (based on TabNet architecture) achieved 82% accuracy, while the vision-based model (features were estimated using deep learning and computer vision) based on Random Forest architecture reached 78% accuracy. A correlation analysis revealed strong alignment between both models’ predictions, with 21 out of 27 sessions showing significant positive correlations on the collected dataset. Further comparison with an earlier-developed vision-based model trained on another dataset supported the generalizability of the vision-based model using physiological indicators extracted from a facial video for fatigue estimation. These findings highlight the potential of the vision-based model as a practical alternative to sensor and special-devices-based systems, especially in settings where non-intrusiveness and scalability are critical.