Credibility Metrics for Student-Assigned Labels of Textual Comments

This research paper describes how pattern recognition can be used to validate student-assigned labels and improve a training dataset natural language processing. In our preceding research, we attempted to enhance peer assessment by harnessing the power of natural language processing (NLP) and machine learning (ML) to critically evaluate the substance and quality of peer-review comments. This required training data, which we had students create by labeling review comments they had received from peers on various aspects of their work. Our previous paper delineated strategies for the automatic validation of labels (“tags”) applied by students, aiming to enhance the reliability of the data fueling our ML algorithms. This paper examines the metrics introduced in our previous work, and studies how effective they are in the evaluation of actual labels assigned by students.