A Multitrait-Multimethod Investigation of Two Checklist Formats for Rating the In-Basket

The two most commonly identified methods of rating the In-Basket both involve the use of checklists. These checklists differ primarily in the way that the items on them are organized. In one (the Dimension Oriented Format), the items pertaining to the entire In-Basket are grouped under the dimension they represent (Frederiksen, Jensen, & Beaton, 1972). In the other (the Item Oriented Format), items representing all dimensions are grouped under the exercise in which they .might be expected to occur (Jaffee, 1971). This study investigated the construct validity of In-Basket ratings obtained using these two formats. The results of an analysis of variance performed on the dimension scores did provide evidence for the construct validity of the two checklists. Effects indicating convergent validity and discriminant validity were both significant while those indicating method bias and measurement and sampling error were not. There were no differences in the convergent validities or discriminant validities of the two formats. Further investigation indicated that the Item Oriented Format enjoyed a higher degree of user acceptance and required approximately 50% less time to use. Based on this evidence, it is suggested that the Item Oriented Checklist might be the best alternative for rating the In-Basket. Intraclass correlation coefficients were compared with those obtained in other -construct validation efforts and possible explanations for the differences were discussed.