Intelligent virtual agents in psychotherapy: a safety evaluation across high-risk mental health scenarios

The growing burden of mental illness and limited access to evidence-based psychotherapy have increased interest in artificial intelligence (AI)-driven conversational agents as potential supports for mental health care. In this exploratory pilot study, we examined the safety and feasibility of an intelligent virtual agent (IVA) designed to simulate psychotherapeutic interactions, with a focus on high-risk situations involving suicidality and substance use. Two licensed psychotherapists engaged in scripted interactions with the IVA across 12 predefined scenarios addressing suicidality and substance abuse. The IVA was powered by GPT-4omni and embedded in a Unity-based avatar. After each interaction, testers evaluated acceptance, usability, and human-robot interaction. Two independent psychotherapists rated the IVA's responses using a structured scale assessing guideline adherence, risk recognition, help provision, de-escalation, and empathy. No real patients were involved; all interactions were simulated for safety testing purposes. The IVA showed preliminary indications of good usability and generally empathic responses. However, problematic responses occurred in 29% of conversations, with 12.5% rated as highly critical. Responses rated as "critical" or "highly critical" referred to outputs that failed to provide adequate support, showed insufficient risk recognition, or included ethically problematic suggestions. Key concerns included inadequate recognition of risk, normalization of substance use, and insufficient referral to crisis resources, particularly in scenarios involving underage alcohol access and suicide-related inquiries. In this small, expert-based pilot safety evaluation, the findings suggest that although AI-based agents may improve access to mental health support, rigorous safety evaluation, clinical oversight, and robust safeguards are essential prior to clinical deployment. No clinical conclusions can be drawn from this simulated study.