Recommendations for Incorporating Large Language Models Into Psychological Research: A Commentary on Austin et al. (2025)

Large language models (LLMs) have clear potential to contribute to psychological research methods. However, as with any new research method, LLM-based methods require rigorous evaluation. Towards this end, Austin and colleagues evaluated the ability of two LLMs (GPT-4o and Gemini 2.0 Flash) to facilitate scale validation by identifying relevant criterion measures and generating validity hypotheses for assessing convergent and discriminant validity. We argue that their study has several key limitations that limit the conclusions that can be drawn about the proposed method's suitability for use in validation research. We situate our critique within three recommendations for incorporating LLMs into research: (a) avoid essentialisation and overgeneralisation about 'AI', (b) use application programming interfaces when evaluating LLMs in research contexts and (c) evaluate LLM-based methods using clearly specified criteria. Finally, we consider whether adopting the proposed method might reinforce or amplify existing limitations in the psychological literature, namely, the proliferation of overlapping constructs and inadequately validated measures.

Authors

Institutions

Publication Details

Journal
Assessment
Published
2026-11-19
DOI
https://doi.org/10.1177/10731911261503860
Primary Topic
Psychometric Methodologies and Testing
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Recommendations for Incorporating Large Language Models Into Psychological Research: A Commentary on Austin et al. (2025)

Beth Clarke, Wendy C. Higgins, Malte Elson, Jamie Cummins
1 links
Assessment
Psychometric Methodologies and Testing
article

Recommendations for Incorporating Large Language Models Into Psychological Research: A Commentary on Austin et al. (2025)

Beth Clarke, Wendy C. Higgins, Malte Elson, Jamie Cummins
article en

Abstract

Large language models (LLMs) have clear potential to contribute to psychological research methods. However, as with any new research method, LLM-based methods require rigorous evaluation. Towards this end, Austin and colleagues evaluated the ability of two LLMs (GPT-4o and Gemini 2.0 Flash) to facilitate scale validation by identifying relevant criterion measures and generating validity hypotheses for assessing convergent and discriminant validity. We argue that their study has several key limitations that limit the conclusions that can be drawn about the proposed method's suitability for use in validation research. We situate our critique within three recommendations for incorporating LLMs into research: (a) avoid essentialisation and overgeneralisation about 'AI', (b) use application programming interfaces when evaluating LLMs in research contexts and (c) evaluate LLM-based methods using clearly specified criteria. Finally, we consider whether adopting the proposed method might reinforce or amplify existing limitations in the psychological literature, namely, the proliferation of overlapping constructs and inadequately validated measures.

Assessment
University of Bern (CH), The University of Melbourne (AU), University of Oxford (GB)
Psychometric Methodologies and Testing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon