Optimizing content validity with randomized item selection in personality measurement

Personality is essential in occupational and organizational research, but measuring it often involves lengthy self-report scales. For instance, the Big Five Inventory (BFI) has 44 items, while the original Minnesota Multiphasic Personality Inventory has 566. To tackle this, researchers commonly create shorter versions. An alternative approach is to assign each respondent a random subset of the full item pool, reducing scale length per respondent while covering all items across the sample. This method offers key benefits: broader construct coverage, less respondent fatigue, minimized item-specific variance, and improved model fit in confirmatory factor analyses (CFA) and structural equation models (SEM). We tested this randomized approach in two studies. The first used simulation data from the LOOPR project (Soto, 2019), assigning respondents (virtually) to six conditions: 2 (3 vs. 6 items) × 3 (fixed, random, full subset). Results indicated superior model fit in the random condition, higher reliability with more items, and comparable predictive validity across conditions. The second study involved 726 participants randomly assigned to four conditions based on two factors: item selection method (fixed vs. random) and the number of items per Big Five domain (3 vs. 6). The fixed-item conditions used three items per domain from the BFI-2 ExtraShort Form (BFI-2-XS) or six from the BFI-2 Short Form (BFI-2-S). The random conditions drew three or six items per domain from a pool of 12 adapted BFI-2 items. Findings confirmed the randomized approach improved content coverage, reduced item-specific variance, and enhanced model fit. While internal consistency was lower, regression coefficients remained comparable, and biases in structural relationships were mitigated. These results suggest that randomized item assignment is a promising, scalable solution to the challenges of scale length and validity in personality research.