Online Learning Platform of Modern Chinese Course Based on Multimodal Emotion-Aware Adaptive Learning

Currently, Deep Learning (DL)-based Intelligent Teaching Systems (ITS) have been widely applied to online Chinese language education to personalize instruction. These systems typically adapt content based on learner performance metrics. However, the existing DL-based ITL systems cannot detect and respond to learners' emotional states in real-time complex language learning. Hence, this study proposes a Multimodal Emotion-Aware Adaptive Learning (MEAAL) framework for proactively adjusting content and support based on learners' emotional and cognitive states. Initially, facial image data were collected and user interaction data were recorded. A Convolutional Neural Network (CNN) with an attention mechanism was employed to extract emotional features. Subsequently, a multimodal fusion model combines facial emotion scores, idle time, and eye-tracking data to calculate the engagement score using weighted averaging. Next, an adaptive learning path decision was handled using a rule-based engine that selected appropriate actions based on the detected emotional state. Furthermore, personalized learning materials are served through a DL content recommendation module, and an updated student model, namely, Recurrent Neural Network (RNN), stores engagement trends as well as emotional feedback for improved future adaptation. The proposed MEAAL model achieved better results in terms of accuracy (96%) than existing DL Dual Neural Network (DL-DNN).