Assessing the Accuracy and Reliability of Large Language Models in Psychiatry Using Standardized Multiple-Choice Questions: Cross-Sectional Study
To our knowledge, this is the first comprehensive evaluation of the general psychiatric knowledge encoded in commercially available LLMs and the first study to assess their reliability and identify predictors of response accuracy within medical domains. The findings suggest that GPT-4 and GPT-4o encode accurate and reliable general psychiatric knowledge and that methods, such as repeated prompting, may provide a measure of LLM response confidence. This work supports the potential of LLMs in mental health settings and motivates further research to assess their performance in more open-ended clinical contexts.
Authors
- Karthik V. Sarma (ORCID: https://orcid.org/0000-0002-7442-9526)
- Kaitlin Hanss (ORCID: https://orcid.org/0000-0003-1462-4335)
- Anne L. Glowinski (ORCID: https://orcid.org/0000-0002-0690-9560)
- Sasha Gorrell (ORCID: https://orcid.org/0000-0002-8861-8547)
- Andrew D. Krystal (ORCID: https://orcid.org/0000-0002-6702-781X)
- Erin E. Reilly (ORCID: https://orcid.org/0000-0001-9269-0747)
- Ramotse Saunders (ORCID: https://orcid.org/0000-0002-6735-4482)
- Andrew J M Halls (ORCID: https://orcid.org/0000-0002-8943-5822)
Institutions
- University of California, San Francisco (US)
Publication Details
- Journal
- Journal of Medical Internet Research
- Published
- 2025-05-20
- DOI
- https://doi.org/10.2196/69910
- Citations
- 5
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 3.27