Effect of Cognitive Pretests on Measurement Invariance and Reliability in Quality of Life Measures: An Evaluation in Refugee Studies

Abstract The issue evaluated was how cross-cultural cognitive pretests affect the reliability and comparability of data when studying refugees and using cross-language comparisons. Three instruments employed to assess general and health-related quality of life were revised based on the findings of cognitive pretests. The versions before and after cognitive pretests were randomly assigned to respondents in two web survey studies. The first study recruited Arabic and Dari speaking refugees ( N = 3448) via Facebook. The second study used a random sample ( N = 610) of Arabic-speaking refugees and the German-speaking population. The comparability of data was evaluated by means of two popular measurement invariance analysis methods (exact and Bayesian measurement invariance analysis). These methods rely on factor analysis and thus enable the evaluation of the comparability of factor structure (configural invariance), the loadings or weights an item has on the factor (metric invariance), and the intercepts of an item on latent factor (scalar invariance). The revision of instruments on the basis of cognitive pretests primarily yielded positive outcomes with regard to reliability and cross-language comparability with respect to the factorial structure and loadings. However, the outcomes pertaining to scalar invariance were more equivocal. Cognitive pretests can therefore help to improve measurement quality, but further research is needed on the cross-cultural questionnaire design to make revisions with respect to cross-cultural comparability more conclusive.