Testing Teacher Judgments Comprehensively: Accuracy, Halo, Frame of Reference, Strategy, and Personality Effects in Holistic and Analytic Assessments of Student Essays

The present study examined teacher judgment accuracy and bias in text assessment. We (a) investigated differences in judgment accuracy depending on multiple text quality criteria, (b) explored teacher characteristics and judgment behavior as potential moderators of judgment accuracy, and (c) juxtaposed accuracy and typical judgment biases, namely reference group and halo effects. A sample of NLevel 1= 6,300 judgments from NLevel 2 = 315 German and Swiss preservice English teachers was analyzed using hierarchical linear models. The results predominantly imply a medium to strong relative judgment accuracy (rank component) across analytic and holistic text ratings. Moderation analyses showed that relative judgment accuracy was higher when teachers spent more time and switched more often between essays to achieve consistent judgments but tended to be lower for teachers reporting more text assessment experience. Language quality and content triggered halo effects in judgments of other text quality dimensions. Mean essay quality had a negative effect on teacher judgments, implying a reference-group effect in line with the contrast hypothesis. Although statistically significant, biases were much smaller than coefficients representing accuracy. We discuss the implications of our findings with regard to teacher education.