Using large language models to predict relationships among survey scales and items

psychological measurement instruments related, and how reliable are they?Here, Hommel and Arslan (2025) trained a language model-SurveyBot3000-to provide answers to these questions efficiently and without human intervention.In their Stage 1 submission, the authors described the training and pilot validation of a statistical model whose inputs are psychological measurement items or scales, and outputs are the interrelationships between the items and scales, and their reliabilities.The pilot results were promising: SurveyBot3000's predicted inter-scale correlations were strongly associated with empirical correlations from existing human data.In this Stage 2 report, the authors further examined the model's performance and validity.In accordance with their Stage 1 plans, they collected new data from 450 demographically diverse participants, and tested the model's performance fully out of sample.The model's item-to-item correlations correlated at r=.59 with corresponding item-to-item correlations from human participants.The scale-to-scale correlations were even more accurate at r=.83, indicating reasonable performance.Nevertheless, the authors remain justifiably cautious in their recommendation that the "synthetic estimates can guide an investigation but need to be followed up by human researchers with human data."The authors documented all deviations between Stage 2 execution and Stage 1 plans in extensive online supplements.These supplements also addressed other potential issues, such as the potential for data leakage (finding the results in the training data) and robustness of results across different exclusion criteria.The authors' proposed psychometric approach and tool, which is freely available as an online app, could prove valuable for researchers either looking to use or adapt existing scales or items, or when developing new scales