Large language models for ADHD clinical label prediction via parent questionnaires: diagnostic performance, explanation groundedness, and failure modes a temporal validation study

Large language models (LLMs) can process descriptions from parent-completed questionnaires, producing risk estimates and auditable reasoning. Psychiatric LLM evaluations have focused on accuracy alone, rarely examining whether explanations are grounded in input data or language behavior changes with errors. We evaluated four LLMs across classification, explanation groundedness, language behavior during errors, and grey-zone boundaries. We analyzed 2,264 de-identified cases from a single-center child psychiatry clinic under temporal validation (training n=1,268, test n=996), with DSM-5 attention-deficit/hyperactivity disorder (ADHD) diagnosis by attending psychiatrists as the reference standard; controls had no other primary psychiatric diagnosis. Four LLMs received descriptions of the Conners Parent Rating Scale and Weiss Functional Impairment Rating Scale. LASSO and XGBoost served as supervised baselines. Rationales underwent rule-based groundedness annotation with expert validation on 80 cases; grey-zone stratification used LASSO composite scores with principal component analysis (PCA). Under the Elevated-Item condition, the four LLMs achieved areas under the receiver operating characteristic curve (AUC) of 0.803–0.823 and Cohen''s κ of 0.470–0.492; Claude Opus 4.7 was within 0.004 of the item-level LASSO baseline (κ=0.488). Rule-based annotation identified 36.2%–41.7% of claims as not directly supported by input data; expert assessment on a subset found ~38% (κ=0.699), matching the automated method. In the grey zone (~38% of cases), all methods performed poorly (LLM κ 0.105–0.140; LASSO κ 0.228), and PCA sensitivity analysis improved but did not eliminate uncertainty. Reasoning structure did not reliably distinguish correct from incorrect predictions. LLMs extracted signals comparable to supervised machine learning while generating auditable rationales. However, more than one-third of their clinical claims were not directly supported by the input data (36.2%–41.7% across models by rule-based annotation, closely matched by expert assessment on a subset at ~38%), language behavior did not reliably signal classification errors, and all methods performed poorly in the grey zone (over one-third of cases), where the simpler, more transparent LASSO baseline was at least as accurate. Because a fluent narrative may present limited evidence as coherent judgment, LLM explanations should be treated as auditable drafts, not diagnostic conclusions. The four-dimension framework offers a reusable template for psychiatric AI evaluation; clinical deployment is not supported.