Large language model diagnostic performance across informant sources in child and adolescent psychiatry: a prospective multicenter study
Large language models (LLMs) are increasingly considered for mental health assessment, yet child and adolescent psychiatric diagnosis depends on information from multiple informants. It remains unclear how the source and richness of clinical information influence LLM diagnostic performance. This study evaluated diagnostic accuracy across child, parent, combined, and clinician-enriched information conditions and compared performance across three widely used LLMs. This prospective multicenter study included 220 children and adolescents aged 1–17 years presenting to two university child and adolescent psychiatry clinics in Turkey. Presenting complaints were obtained separately from children and parents. Four input conditions were created: child only, parent only, child plus parent, and child plus parent supplemented with physician-derived clinical information. Three LLMs were evaluated using independent single-turn API queries, followed by two repeat runs to assess reproducibility. Clinician-established DSM-5 diagnoses served as the reference standard. Performance was assessed using Top-1 and Top-3 accuracy, mean reciprocal rank, Cohen’s kappa, F1 scores, and sensitivity and specificity for detecting any psychiatric diagnosis. Models and conditions were compared with paired tests and generalized estimating equation models. Top-1 accuracy was lowest in the child-only condition (35.5–44.1% across models), increased to 65.9–69.1% with parent information, and was highest when physician-derived information was added (78.6–83.2%). Adding the child narrative to the parent narrative did not significantly improve accuracy (69.1–71.8%). Full-information Top-3 accuracy ranged from 88.2 to 91.8%. Model differences were significant only in the child-only condition. With parent narratives, sensitivity for any psychiatric diagnosis exceeded 93%, but specificity was 14.3–42.9%; physician-derived information increased specificity to 52.4–61.9%. Age was associated with higher accuracy only in the child-only condition (OR = 1.164 per year, 95% CI 1.071–1.264). Two repeat runs reproduced the main information-condition findings; between-model differences were less consistent. LLM diagnostic performance varied with the available clinical information, with the highest accuracy observed when physician-derived findings were included. Adding child narratives to parent reports did not significantly improve accuracy, and the models frequently assigned diagnoses to children without a psychiatric disorder. These observational findings are based on routine clinician-established diagnoses and do not demonstrate improvements in clinical decision-making or patient outcomes.
Authors
- Hakan Kayış (ORCID: https://orcid.org/0000-0001-6800-9587)
- Akın Tahıllıoğlu (ORCID: https://orcid.org/0000-0002-3952-3672)
- Ayşegül Tuğba Hıra Selen (ORCID: https://orcid.org/0000-0003-1065-5548)
- Çınar Gedizlioğlu (ORCID: https://orcid.org/0000-0003-3288-6080)
Institutions
- İzmir University of Economics (TR)
- Zonguldak Bülent Ecevit University (TR)
- Bakırçay Üniversitesi (TR)
- Konya City Hospital (TR)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-20
- DOI
- https://doi.org/10.1038/s41598-026-81830-1
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00