The WEIRD instrument problem and systematic bias in cross-cultural research: the example of personality psychology

As researchers broaden samples to populations that are not Western, Educated, Industrialized, Rich, and Democratic (WEIRD), they often use instruments, such as questionnaires, designed for WEIRD populations. Does this practice create systematic biases when making cross-cultural comparisons? Inspired by the example of less apparent structural complexity in personality data outside of WEIRD populations, I show with computational models that designing a personality instrument for a reference, typically WEIRD, population and applying it to a novel population will systematically underestimate the relative complexity of the novel population. This occurs because instruments optimized to capture the personality structure of one society fail to capture alternative structures in other societies. The models also make the novel quantitative prediction that a society's "cultural distance" from the reference society should tend to be negatively associated with the apparent complexity of its personality structure. To test this prediction, I used pre-existing cross-cultural personality and cultural distance datasets to show that cultural distance from the United States, from which the reference personality instrument was designed, has the strongest negative relationship with the complexity of the personality structure, as measured by the instrument, out of all 67 countries in the dataset. This result duplicates an outcome of the model. These findings suggest that lower apparent complexity in non-WEIRD groups may be driven by the choice of WEIRD instruments, rather than by actual differences in complexity. These biases would also tend to occur in any case where an instrument designed for one group is applied to another, such as different genders or cohorts. I propose possible reforms to avoid this type of bias.