Analysis of Computational Approaches to Cognitive Diagnosis

Access to good education is crucial to the well-being of individuals as well as communities. Recent technological advancements in the field of computer science show promise of generating precise descriptions of student cognitive states regarding specified knowledge concepts through a process called cognitive diagnosis. This can facilitate the creation of more targeted lesson plans and more personalized educational software. Experiments were conducted to evaluate the performance of four computerized cognitive diagnosis models. The models include three existing models: Item Response Theory, Neural Cognitive Diagnosis, Knowledge Association Neural Cognitive Diagnosis, and a proposed model, Concept Agnostic Knowledge Evaluation, which was used to quantify the value of labeling exercises with knowledge concepts. These models were trained and evaluated using four separate data sets: the 2009 ASSISTments data set, the NeurIPS 2020 dataset, the Junyi 2015 data set, and the ASSISTments 2012 dataset. Each model-data set combination was evaluated under five distinct conditions: "basic" and "sampled" to determine the effect of data sampling, as well as "undersampleEven," "undersampleCorrect," and "undersampleCorrect," and "undersampleIncorrect" to determine the effect of manipulating the class ratios. The Accuracy, Area under the ROC curve, mean average error, and root mean squared error of each experiment was recorded. Results reaffirmed the superiority of neural network models over psychometric models. Concept labeling was found to greatly improve interpretability with comparable performance results. Data sampling was shown to be an effective means of improving neural network training time. It was shown that model performance may degrade as user knowledge improves. Future research may investigate ways to adapt to changing user knowledge states, the development of automatic question labeling, or question generation from labels to further facilitate the collection of data on the cognitive diagnosis task.