When conviction outpaces evidence: diagnostic boundary failures of large language models in adult ADHD self-assessment

Large language models (LLMs) are increasingly used by people seeking information about possible mental-health conditions, including attention-deficit/hyperactivity disorder (ADHD). This raises an important safety question: can these systems distinguish between stronger clinical evidence and a user’s increasing confidence in a self-diagnosis? We evaluated ChatGPT, Gemini, and Doubao using 24 controlled adult ADHD scenarios representing presentations compatible with ADHD, presentations with insufficient evidence for diagnosis, and presentations with plausible alternative explanations. For each scenario, the models were tested as either clinically relevant information accumulated or the user’s confidence in having ADHD increased without corresponding diagnostic evidence. Testing was repeated independently, producing 1,440 responses. Models were more likely to express unsupported diagnostic certainty when user confidence increased without stronger clinical evidence than when clinically informative evidence accumulated. This pattern was most apparent when the available information was insufficient for ADHD diagnosis or when other explanations remained plausible. In contrast, when clinically relevant information was progressively introduced, model responses generally changed in the direction supported by that information. The three models also differed in their overall diagnostic safety. These findings suggest that evaluating mental-health LLMs requires more than assessing factual accuracy. Safe conversational systems should preserve appropriate diagnostic uncertainty and avoid allowing increasing user confidence alone to substitute for clinical evidence.