The Effect of Rater Discussion on Rating Characteristics in Korean Writing Assessment

The purpose of this study is to analyze the effect of rater discussion on rating characteristics in Korean writing assessment and to propose a practical method for ensuring scoring reliability that can be applied in Korean language education settings. A group of 29 raters participated, initially scoring 15 writing samples independently, then engaging in group discussions to resolve scoring discrepancies, followed by scoring another set of 15 samples. The Many-Facet Rasch Model (MFRM) was used to evaluate changes in rater characteristics, including severity, consistency, and bias. The findings showed that rater discussions effectively narrowed the range of severity, leading to a more centralized distribution of scores. Consistency also improved, with fewer raters displaying inconsistency in the second round. Moreover, the occurrence of significant bias diminished, suggesting that discussions helped mitigate tendencies toward leniency or harshness. These results indicate that rater agreement not only enhances rater expertise but also contributes to establishing a shared understanding of evaluation criteria among raters.