A Comprehensive Analysis of Data Imbalance in Deep Learning-Based Cognitive Diagnosis

Cognitive diagnosis aims to quantify students’ learning status and mastery level of related knowledge concepts based on their responses to given exercises. This is a fundamental yet critical research task in the field of intelligent education, which helps to reveal students’ proficiency on multiple knowledge concepts they have learned, thereby providing personalized learning services for each student. In recent years, researchers have succeed in improving model’s diagnosis accuracy by designing diagnostic functions based on deep neural networks or integrating richer contextual features to enhance the representation learning of students and exercises. However, datasets used for deep learning-based cognitive diagnosis model training often present an imbalanced distribution, being a large number of students only answered a few exercises, and a large number of exercises were answered by only a few students, which may have a certain impact on the performance of the model. To verify this problem, we conducted considerable experiments on four well known models and two widely used datasets of deep learning-based cognitive diagnosis in this paper. Firstly, we analyzed the correlation between the model’s predictive accuracy for individual student’s response performance and the number of exercises answered by this student. Secondly, we studied the correlation between the model’s predictive accuracy for each exercise and the number of the exercise being answered by students in the dataset. Finally, we analyzed whether the model’s predictive accuracy for individual student would be over-fitting<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> during multiple epochs and whether the maximum predictive accuracy achieved is affected by the number of exercises answered by this student. The experimental results indicate that there are no evident statistics supporting the strong correlation between the model’s predictive accuracy for individual student and the number of exercises answered by this student. The same case happens with the correlation between the model’s prediction accuracy on each exercise and the number of the exercise being answered by students in the dataset. Notably, we observe that models are more likely to be over-fitting for students who have answered a larger number of exercises.