Multitrait-Multimethod Analysis of Performance Ratings Using Behaviorally Anchored and Behavioral Checklist Formats

The construct validity of a behaviorally anchored rating scale (BARS) and a behavioral checklist was examined for evaluation of performance in leaderless group discussions. The assessors consisted of 12 graduate students in industrial/organizational psychology who were randomly assigned to use the behavioral checklist or the BARS format. Performance ratings were made of 12 ratees in both assigned and nonassigned role leaderless group discussions. The ratings were analyzed with a multitrait-multimethod approach using analysis of variance (Dickinson 1977; 1984) ~ Convergent validity and discriminant validity were found for the assigned discussion, whereas only discriminant validity was found for the nonassigned discussion. Low method bias was found in the ratings for both discussions. Comparisons of the two formats showed the checklist to yield higher discriminant validity for the nonassigned discussion while the BARS had higher validity for the assigned discussion. These findings suggested that different formats are appropriate for evaluating performance in different exercises. Additional research was recommended for improving leaderless group discussions and for determining appropriate formats'