Probabilistic Graph Models for Peer Assessment Based on Student Grading Ability Under the Setting of Spot-checking

With the proliferation of many MOOC platforms, grading open-ended assignments submitted by many students presents a significant challenge in educational research. Peer assessment, which requires students to act as peer graders and evaluate their peers’ submissions of assignments, is the mainstream solution to address this issue. Researchers have recently proposed various probabilistic graph models to evaluate peer graders’ grading reliability and bias, effectively improving the estimated actual scores of assignments based on peer grades. However, the existing probabilistic graph models consider only the impact of students’ scores on the current assignment regarding their grading reliability, failing to account for their scoring deviation, which directly measures their reliability. This limitation affects the performance of these models. Therefore, this study proposes two novel probabilistic graph models, RPG₆ and RPG₇, which incorporate the peer graders’ grading ability, quantified based on their score deviation within a small proportion of submissions being spot-checked by teachers. These models, constructed on the foundation of two existing probabilistic graph models, represent the grading reliability of peer graders as a variable dependent on their scoring deviation-aware grading ability rather than their scores for the current assignment. This approach enhances the estimation of the true scores of assignments. Real classroom experiments demonstrated that the proposed RPG₆ and RPG₇ models achieve greater accuracy in estimating the true scores of assignments in peer assessment activities. Specifically, the RMSE values of RPG₆ and RPG₇ are, on average, 11.75% lower than those of the state-of-the-art method.