Convergent Validation of Trait Induction in LLMs

Abstract: This study investigates an approach to test a key requirement for using large language models (LLMs) to simulate valid data sets of item-responses, namely the effect of trait induction on another related construct. Using OpenAI's GPT-4, we used zero-shot prompting to induce high- and low-agreeableness profiles in the model's output. We then investigated, in terms of convergent construct validation, the effect of trait induction on another theoretically and empirically related construct: emotion understanding (EU). To measure EU, we utilized a situational judgment test developed to test LLMs. Using n = 680 simulated data sets, we show that a prompt on a trait (agreeableness) has a plausible effect on unprompted behavior in a skill (EU). These results provide initial support for evaluating the effects of trait induction through methods of convergent construct validation when assessing LLM-generated responses, and for using LLMs to generate data for foundational steps of psychometric test development.