The Effects of Training and Rating Formats on the Accuracy of Performance Ratings

The effects of frame of reference training and rating formats on the accuracy of performance ratings were examined utilizing direct measures of accuracy in addition to commonly used measures of rating biases. Participants were 111 undergraduate students who were randomly assigned to one of six treatment conditions formed by crossing two levels of training (training, no-training) with three rating formats (graphic rating scale, behaviorally anchored rating scale, mixed standard scale). Ratings were made of the videotaped performance of interviewers conducting selection interviews. These ratings ~ere compared with the mean ratings of "expert raters" to give measures of rating biases (leniency/severity, halo) and direct measures of accuracy (correlation accuracy, distance accuracy, differential accuracy). For training, the results indicated that distance accuracy was the only measure for which there was a significant effect. For the formats, the graphic rating scale and mixed standard scale had significantly less leniency/severity than the behaviorally anchored rating scale, the graphic rating scale had significantly less halo than the other formats, and both the graphic rating scale and behaviorally anchored rating scale formats were more accurate than the mixed standard scale on the correlation accuracy measure. In addition, significant training by formats interactions were found on each of three post-experimental questionnaire items which assessed format preference. It was concluded that the training was not effective. Suggestions were made for improving the frame of reference training approach. For the formats, it was concluded that, at least with inexperienced raters, the graphic rating scale format produced the most accurate ratings. Finally, this study suggested that preference for the behaviorally anchored rating scale format, as defined by the rater's perception of the format's achievement of accurate ratings, clarity, and ease of use, was significantly better with training.