Human Vulnerability, Machine Law: Ethical Risks and Global Governance Challenges in Generative AI

Generative Artificial Intelligence (GenAI) is rapidly transforming digital engagement, but its expanding capabilities introduce novel ethical and legal dilemmas. This paper investigates the under-examined risks of using GenAI models as informal life coaches and therapists, in addition to the other notable differences in regional AI governance, especially in Asia and Africa. Utilizing a mixed-methods design, this study integrates AI simulation testing of four leading models (ChatGPT, Gemini, Claude, and Grok) against the Columbia Suicide Severity Rating Scale (C-SSRS) combined with survey data from 592 users about using AI as life coach or therapist. Results indicate a difference between the AI models which prioritise safety; specifically, the AI model either disengaged from, or provided repetitive, or generic advice, when assessing a high-risk situation where a user might harm themself. The survey also questions the usefulness of AI chatbots—where most users stated that they were “not at all helpful” (55.4%) in providing emotional support , users were “never better ” (56.7%) after using AI as a therapist and prefer human therapists (79.2%) for privacy reasons. Further, the analysis suggests a clear global split in policy; with the EU providing a proactive regulatory framework as opposed to legal and enforcement gaps in the Global South. Overall, this research suggests the use of GenAI for mental health support is premature and facilitates significant risk to vulnerable users. Finally, this paper argues for urgent, risk-based regulations, hybrid models of AI and human support and user-centred research, to create ethical guardrails for responsible innovations in AI.