Psychometric Controversy Involving Job Performance Ratings

Job performance ratings by supervisors and peers are very often the procedure for performance appraisal. The reliability of performance ratings is an important issue, because performance appraisal serves several purposes in the context of human resource management. Unfortunately, there are performance rating biases connected with raters, ratees, the interaction of raters and ratees, and situational and organizational characteristics. Interrater correlations are frequently interpreted as estimates of the reliability of performance ratings, and are also used to correct the correlations between ratings and other measures for attenuation. Some researchers (Murphy & DeShon, 2000a; 2000b) argue that interrater correlations are not reliability coefficients and should not be used to correct for attenuation, because job performance ratings reflect multiple sources of error and fail to meet appropriate assumptions. Others researchers do not agree (Schmidt, Viswesvaran, k. Ones, 2000), maintaining that interrater correlations are an appropriate measure of the reliability for job performance ratings. In addition to this controversy, these researchers also disagree on the relationship between reliability and validity of job performance ratings. The purpose of the current study was to investigate these arguments using simulated data and manipulating factors such as predictor interrater reliability, validity, and performance interrater reliability. The results indicated that the corrected validities provided by the classical test theory (or SVO) and G-theory procedures were virtually identical to those obtained with the SEM procedure. The corrected validities for the MD1 and MD2 conditions were relatively smaller in magnitude compared to those for SEM, SVO, and G-theory. Lastly, correction for attenuation was implemented with an SEM analysis for MD1 and MD2. These structural equation modeling results were identical to those obtained with the SVO conditions for SEM, SVO, and G-theory. Clearly, SEM is a powerful analytic tool that correctly estimates true validity even in the presence of rater idiosyncrasy. Limitations and suggestions for future research are discussed.