Construct Validation of Role Playing Exercises in an Assessment Center Using BARS and Behavioral Checklist Formats

The purpose of this study was to establish the construct validity of five dimensions, assessed by two role simulations, using behavioral checklist and BARS formats. Participants were 12 graduate students in psychology, most of whom were enrolled in an industrial/ organizational program. They served as assessors and evaluated the videotaped performance of ten ratees who had previously participated in assessment centers conducted at the university. Substantial evidence for convergent and an adequate level of discriminant validity were hypothesized. Although it was anticipated that method bias would be low, it was hypothesized that the behavioral checklist would exhibit a higher level of discriminant validity than the BARS format, because the latter format required greater cognitive processing. The analysis of variance approach was utilized in the analysis of the ratings. Significant F-ratios were obtained for convergent and discriminant validity, method bias, and differential discriminant validity with regard to format and exercise. As these ratios were based on mean squares with large degrees of freedom, variance components and intraclass correlation coefficients were also calculated. In contrast to the F-ratios, these latter statistics proved to be quite small for both convergent validity and method bias, accounting for three and one percent of the variance respectively. Thus, there was no support for the prediction made with regard to convergent validity, although the hypothesis concerning method bias was upheld. A moderate level of discriminant validity was attained, accounting for 20 percent of the variance, and thus confirming the hypothesis regarding this effect. The intraclass correlation coefficients associated with differential discriminant validity were .05 for the formats and .18 for the exercises. Post hoc tests revealed that the BARS format resulted in a larger number of significant differences between ratees on four of the five dimensions, while equal numbers of significant differences resulted on the fifth dimension. It was suggested that the large number of behavioral items on the checklist may have made rating with this format more difficult than with the BARS. Even though both of the exercises were role simulations, a pronounced exercise effect was uncovered. The impact of the exercises may have accounted for the low levels of convergent validity, as suggested by Neidig and Neidig (1984). Furthermore, the variance explained by the Exercises x Dimensions x Ratees interaction may have depressed the results with regard to discriminant validity. Additional work is needed to correct the deficiencies noted with regard to the behavioral checklist. Of greater importance, however, is the need to direct more attention to the impact of the exercises on the performance of participants in assessment centers and to examine more closely the manner in which the exercises interact with the dimensions. Further study in these areas may ultimately lead to more satisfactory findings with regard to convergent and discriminant validity.