AI Psychometrics: Evaluating Measurement Invariance in Science Self-Efficacy Using Large Language Models

The present study (1) examined sex- and language-based measurement invariance of the science self-efficacy scale using PISA 2006 and 2015 U.S. data (N = 11,323) and (2) evaluated six large language models (ChatGPT, DeepSeek, Grok, Copilot, MetaAI, Gemini) in generating invariance approximations from prompt content and pre-existing model knowledge. Multigroup confirmatory factor analysis (MG-CFA) established empirical invariance benchmarks across sex and language groups. The same six LLMs were prompted to generate plausible fit-index values (CFI, RMSEA) and invariance decisions without access to empirical response data. Approximation discrepancy and agreement with the MG-CFA benchmarks were assessed using RMSE, MAE, Cronbach’s α and intraclass correlation coefficients (ICCs). MG-CFA supported full measurement invariance across sex and language groups in both cycles. In contrast, LLMs showed strong apparent consistency for CFI (α = 0.857) but poor convergence with empirical data (single-measure ICC = 0.302 for CFI, 0.024 for RMSEA). Discrepancies were often large enough to affect invariance interpretations and the models often overestimated model fit while producing invariance decisions that were not always aligned with the empirical MG-CFA benchmarks. Findings support the science self-efficacy scale’s cross-group validity but caution that current off-the-shelf LLMs, when prompted to generate plausible psychometric approximations without empirical data, produce unreliable outputs for high-stakes psychometric decisions. Implications, limitations and future directions are discussed.