Analyzing the Effectiveness of Reflection Prompts Accompanying Cybersecurity Assignments Using Natural Language Processing

This research paper describes the use of natural language processing to categorize reflection responses according to their level of learning. This could help instructors in assessing the effectiveness of the prompts in encouraging meaningful reflection and students' engagement with the problem. Reflective practice is the process of using one's beliefs and prior experiences to analyze a problem; it is making meaning from experience. Answering reflection prompts has been shown to aid students in learning problem solving skills. This paper describes results of an experiment in which a pretrained bi-directional encoder representations for transformers (BERT) model was used to classify responses to reflection responses. In an earlier experiment, a method was developed for deductively coding prompts using four progressive levels of learning derived from Dewey and Moon: Noticing, Making Sense, Making Meaning, and Transformative Learning. Responses to reflection prompts designed to encourage different levels of learning were manually coded. In the experiment described in this paper, manually coded responses were used to fine tune models that classify reflection prompts for the presence of each level of learning. The models performed well on the test set, indicating substantial agreement with the assigned manually codes. The models were then used to classify responses to reflection prompts accompanying assignments in on input validation vulnerabilities in two Computer Science courses. Three reflection prompts meant to encourage successive levels of learning were given with the assignments. The models proved to be effective in classifying responses, and gave insight into the effectiveness of the reflection prompts, and gave insight into students' engagement with the material in two different classes.