Evaluating Psychological Scales with Gold-Standard Items: Lessons from Measures of Body Height, Weight, and Age

This article investigates whether gold-standard items are helpful for questionnaire development. Based on an initial item pool (138 items) derived through a deductive item construction (𝑁 = 124) and a prototype approach (𝑁 = 24), followed by cognitive interviews (𝑁 = 8) and a pilot study (𝑁 = 390) of a preliminary item set (61 items), three 12-item scales were constructed to measure the physical traits of body height, body weight, and age. We collected data on these scales using response formats with either two (𝑁 = 921) or six (𝑁 = 933) categories. We also collected numeric self-reports of body weight, body height, and age as gold-standard items. We argue that the items intended to measure the latent variable (i.e., body height, body weight, age) most directly (i.e., the gold-standard items) should have the highest loadings on that variable. Contrary to this assumption, confirmatory factor analyses revealed that the gold-standard items did not consistently exhibit the highest loadings on their corresponding latent variables, calling into question the content validity of our scales. Furthermore, when controlling for the self-reported physical body height, body weight, and age as gold-standard items, as well as gender, we did not always find an interpretable, systematic residual variance. Finally, the pattern of correlations among the latent variables did not closely reflect those among the self-reported gold-standard items, suggesting that the item scales and the gold-standard items do not share the same validity. While these results are consistent with previous studies, our analyses also showed that items with two response categories were as valid as those with six categories, contradicting past recommendations. Our analyses underscore the importance of intensive cognitive pretesting in scale construction and suggest that questionnaires using different response formats should be compared empirically and that hypotheses about which items best represent the latent variable should be tested.