Exploring generalizability from the bottom up: Applying DTS to demonstrate an item-level approach to personality assessment

• Exploring invariance across groups mostly involves scale-level comparisons. • This study uses a new statistical test to explore item-level differences. • This ECDF-based statistic provided comparisons in responding across 13 nations. • Nation pairs vary greatly in how much they differ in responding across items. • Some Openness-related items demonstrated few differences across nations. Evidence for measurement invariance across groups is regarded as a prerequisite for administering a personality measure in diverse samples and an important criterion for generalizability. However, popular statistical techniques for establishing measurement invariance focus predominantly on broad, scale-level constructs, and there is growing recognition among personality psychologists of the value of item-level assessment. This poses a challenge: while the higher degree of precision offered by more narrowly defined constructs can be beneficial for advancing our scientific understanding of topics studied in personality psychology, it can come at the expense of conducting culturally diverse and inclusive studies. This paper explores the role of measurement invariance evidence in studies that focus on narrower personality constructs––particularly at the item-level––using a sample from the Synthetic Aperture Personality Assessment (SAPA) project consisting of participants ( N = 1025,954) residing in 13 culturally and geographically diverse nations. We address the overall lack of available statistical techniques for addressing invariance-related questions at the item-level and demonstrate an approach that compares the item-level empirical cumulative distribution functions (ECDFs) of two groups using 2623 items from the International Personality item Pool (IPIP). Findings point to nations and items with the most significant differences in ECDFs, which may reflect true cultural differences or potential psychometric challenges. ECDF-based tests capture distribution-wide differences that similar, more standard tests cannot detect, making them potentially useful for addressing a variety of research questions.