Evaluating Ratings from Large Language Models: A Simulation-Based Workflow

Large Language Models (LLMs) are increasingly used as research tools to facilitate fast and automated ratings of text features. In psychological studies, they have been used to quantify the degree to which stimulus materials reflect certain psychological constructs. However, the application of LLMs entails a high degree of flexibility regarding prompt design (e.g., instruction details and examples) and model specification (e.g., model family, size, and configuration), which can produce divergent results and threaten the robustness and generalizability of conclusions. To navigate the multiverse of possible choices, we develop a systematic workflow for evaluating the quality of LLM-generated ratings across diverse model and prompt specifications. Motivated by generalizability theory, the workflow distinguishes between construct variance across stimulus items, method variance due to model and prompt choices, and residual variance across repeated iterations. To guide researchers through the planning, execution, and reporting of LLM simulation studies, we adapt the ADEMP (Aims, Data-generating mechanism, Estimands and targets, Methods, Performance measures) framework, originally developed for methodological simulation research. We develop a template supporting two complementary validation strategies: assessing consistency across LLM specifications based on generalizability theory via estimated variance components, and external validation against a benchmark (e.g., human ratings). We illustrate the workflow in a preregistered case study using local, open-weight LLMs which examines the influence of model choice, response format, and prompt examples on the quality of valence and arousal ratings for multi-word expressions.