Evaluating Construct Convergence: A Canonical Correlation Analysis of Diagnostic Classification Model-Based SJT and IPIP-NEO-PI-120

The Five-Factor Model (FFM) is the most widely used framework for understanding personality, typically assessed through self-report inventories such as the IPIP-NEO-120. Although these tools show strong psychometric properties, reliance on self-reports introduces vulnerabilities to bias, including social desirability, response distortion, and faking, particularly in high-stakes contexts like personnel selection. To address such limitations, researchers have increasingly explored behavior-oriented measures, notably Situational Judgment Tests (SJTs). When analyzed using Cognitive Diagnostic Models (CDMs), SJTs can yield multidimensional mastery profiles that offer a nuanced representation of personality. However, limited evidence exists regarding the convergence between CDM-based SJT profiles and traditional self-reports. This study applied Robust Canonical Correlation Analysis (RCCA) with a Spearman rank correlation matrix to examine structural alignment between CDM-based SJT scores and IPIP-NEO-120 domain scores in a sample of 289 participants. RCCA was chosen to handle common violations of multivariate assumptions, such as non-normality and outliers. Results showed a moderate canonical correlation in the first root, supporting convergent validity while highlighting distinct trait representations across methods. Subsequent roots yielded weaker and less stable associations. Overall, findings indicate that RCCA is a viable approach for evaluating cross-method validity and emphasize the complementary insights of situational and self-report approaches in personality assessment.