Integrating expert knowledge into large language models improves performance for psychiatric reasoning and diagnosis
When used to predict psychiatric diagnoses from case vignettes, direct prompting of the LLMs yielded most true positive diagnoses but had significant overdiagnosis. Integrating expert-derived reasoning into the process using decision trees improved LLM performance (as measured by F<sub>1</sub> statistic), primarily by suppressing overdiagnosis with a lower-magnitude negative impact on sensitivity. This suggests that the integration of clinical expert-derived reasoning could improve the performance of LLM-based tools in the behavioral health setting.
Authors
- Daniel J. Becker (ORCID: https://orcid.org/0000-0003-4315-8628)
- Karthik V. Sarma (ORCID: https://orcid.org/0000-0002-7442-9526)
- Kaitlin Hanss (ORCID: https://orcid.org/0000-0003-1462-4335)
- Anne L. Glowinski (ORCID: https://orcid.org/0000-0002-0690-9560)
- Atul J. Butte (ORCID: https://orcid.org/0000-0002-7433-2740)
- Andrew D. Krystal (ORCID: https://orcid.org/0000-0002-6702-781X)
- Andrew J M Halls (ORCID: https://orcid.org/0000-0002-8943-5822)
Institutions
- University of California, San Francisco (US)
- University of San Francisco (US)
Publication Details
- Journal
- Psychiatry Research
- Published
- 2025-11-20
- DOI
- https://doi.org/10.1016/j.psychres.2025.116844
- Citations
- 2
- Primary Topic
- Mental Health via Writing
- Type
- article
- Field-Weighted Citation Impact
- 4.88