Beyond experience: Rating quality and rater behaviors in integrated writing

Assessing source-based writing is challenging for raters due to the complexity of language usage and source integration (Plakans & Ohta, 2021). During the rating process, raters draw upon their prior experience, but despite a substantial body of research (Barkaoui, 2010a; Lim, 2011), no consensus has been reached regarding the impact of experience on assessment quality. This explanatory mixed-methods study compares rating performance of six novice and four experienced raters, who rated text products from integrated writing tasks using an analytic rating scale. The quantitative strand examines differences between the groups in terms of interrater agreement, severity, and halo effect. For the qualitative strand, think-aloud protocol and focus group interview data were collected. A set of qualitative variables that could help to explain the quantitative findings was identified based on prior research (e.g., Crisp, 2008; Zhang, 2016; Wolfe et al., 1998). Concerning interrater agreement and severity, the study results indicate greater variance between individual raters than between groups. In the case of the halo effect, we found that experienced raters were more susceptible to this effect than novices. The findings enhance our understanding of the role of rater experience and rater training for scoring approaches and rating quality in the context of integrated writing assessment.