A Multimodal Deep Learning Approach to Decoding Student Participation in Online Classrooms

The COVID-19 has accelerated the e-learning usage, stressing a proper research of how students engage themselves in online classes. Previous works are mainly concerned with uni-modal treatment where data from only one modality such as facial expressions or eye-gaze is analyzed which provides a very narrow perspective of engagement. Towards this, we introduce a novel multimodal deep learning framework fusing facial expression and eye gaze, textual chat and interaction transcripts, along with contextual data on learning. This framework uses deep learning models for analysing multimodal data in real time so that timely interventions and feedbacks can be given to each learner. It also improves interpretability by including attention mechanisms, which show the level of importance of the different modalities and features in producing engagement estimates. We also show through real-world experiments on a carefully selected benchmark set the effectiveness of the presented framework with respect to unimodal as well as the state of the art multimodal techniques. And not only that, it is capable of identifying engagement and also giving instructional suggestions to help educators modify their course and make learning more engaging. This research opens the way to the creation of adaptive e-learning systems that could enhance student-centered online learning environment.