Detecting Suicidal Ideation Signals in Synthetic Conversations with an LLM-Based Mental Health Chatbot: Experimental study

Background: Large language models (LLMs) are increasingly being investigated as tools for supporting mental health. To ensure user safety, they require strategies to recognize possible risks such as suicidal ideation and capabilities for appropriate reaction. Objective: This study evaluated the performance of an LLM-based chatbot in accurately stratifying suicidal ideation among university students and to provide appropriate responses. Methods: Five variants of the system prompt were designed to classify suicide risk and produce suitable responses in context. The best-performing prompt was selected through expert evaluation using a standardized test dataset, refined, and subsequently tested with four LLMs (GPT-5, Gemma3:27b, Deepseek-r1:32b, and Qwen3:32b) using 100 synthetic user inputs. Results: Across all model variants, suicide risk classification performance across seven risk levels was generally low, with weighted-average F1 scores ranging from 0.23 to 0.34. GPT-5 achieved the highest accuracy in both exact and near-exact (within one risk class) classifications. However, Gemma3:27b demonstrated the strongest correlation with human ratings (Spearman’s ρ = 0.73–0.83), suggesting consistent relative judgment. Qualitative analysis showed that 64% of chatbot responses were rated as appropriate or very appropriate, and none were judged potentially harmful. Conclusions: Despite the responses generally being non-toxic, none of the models reliably stratified suicidal ideation, indicating a critical safety gap. Underestimation of risk and the subsequent occurrence of false negatives could result in delayed intervention. Further work is required to optimize the process promptly, incorporate conversational context and use complementary assessment methods before LLMs can be used safely in mental health settings.