When should mental health chatbots have a duty to report?
Abstract People increasingly disclose suicidal intent, child abuse and threats of violence to AI chatbots. Made to a clinician, such disclosures would trigger mandated-reporting statutes, Tarasoff -style duties to protect third parties and a duty of care toward patients at risk of self-harm. Made to a chatbot, they trigger almost none. The first chatbot-specific laws, in California and New York, require crisis referrals but impose no duty to report or intervene. We argue that a chatbot’s legal duties should follow the function it holds itself out as serving. We distinguish FDA-regulated mental-health devices, chatbots marketed as therapeutic or emotional support, and general-purpose conversational AI. For the middle category we propose five duties: detection of defined risk indicators, human review before external escalation, proportionate crisis response, independent audit and record-keeping. We then examine enforcement through market-access statutes and litigation, and the confidentiality questions any reporting regime raises.
Authors
- Sofia Palmieri (ORCID: https://orcid.org/0000-0002-6618-5963)
- Ivan Glenn Cohen (ORCID: https://orcid.org/0000-0002-0223-3305)
- Vincent Joralemon (ORCID: https://orcid.org/0009-0007-6115-7836)
Institutions
- Harvard University (US)
- University of California, Berkeley (US)
Publication Details
- Journal
- Research Square
- Published
- 2026-10-29
- DOI
- https://doi.org/10.21203/rs.3.rs-11155348/v1
- Primary Topic
- Digitalization, Law, and Regulation
- Type
- preprint
Funders
- Novo Nordisk