Occlusion-aware Data Augmentation and Reconstruction for Improving Student Engagement in Online Learning Environments

The demand for adaptive learning in education is an important application for Computer Vision (CV)-based detection models which extract learners’ faces to classify engagement. However, a loss in visual data due to the partial occlusion of learners’ faces imposes challenge for practical use-cases. In this paper, we propose an occlusion-aware framework to improve the robustness of non-occlusion-aware models in engagement detection. Firstly, our framework consists of an occlusion-aware data augmentation pipeline that aims to simulate the types of partially occluded faces in-the-wild. Secondly, we investigate the application of masked auto-encoders (MAE) for occlusion recovery. Extensive experiments have been performed to measure the effectiveness of the proposed occlusion-aware framework. Specifically, on the challenging FER-2013 dataset our occlusion-aware data augmentation achieves a 5.24% improvement in accuracy on baseline models and a 1.54% improvement against models with augmentation methods used in the state-of-the-art. Alongside the application of MAE, our proposed framework improves baseline classification accuracy by 13.3% and 4.59% on the FER-2013, and DAiSEE dataset for engagement detection. Our investigation into occlusion-aware engagement detection using the DAiSEE dataset provides important insight into the limitations of occlusion-awareness in engagement detection.