Designing AI for distress: an iterative case study of a hierarchical agent for psychosocial support

Introduction: In recent years, psychosocial issues like bullying and ragging have risen all around the world. Even though there are some victims who report such cases, most of them never do so because of fear, stigma, or mistrust, mainly among students. This scenario highlights the necessity of safe, anonymous, and easily available support system. The promising approach to solving this problem is conversational AI because it allows people to request assistance in a simple and an easy-going way. However, conversational AI systems often lack contextual sensitivity, emotional intelligence, and appropriate support response mechanisms. To address this, there is a need for structured frameworks integrating human-in-the-loop oversight along with preventive and actionable insights to ensure ethical and proportional responses. This paper introduces SocialWellbeing, a socio-technical system which is designed to overcome psychosocial harms in a three-pillar framework. Methods: The initial pillar offers artificial intelligence-based first-line care whereby users can share their issues anonymously and get caring responses. The second pillar facilitates human-guided escalation where more serious cases are passed to trusted advisors or authorities to provide real-life support. The third pillar is devoted to data-driven prevention, where anonymized data are used to identify trends and serve as a guide in the institutional preventive efforts. Results: The AI system was tested in two stages. Phase 1 involved prompt engineering that identified the types of incidents, detected the emotions, and generated supportive responses. Phase 2 improved this strategy with a chain-of-thought strategy, which was A.G.E. (Acknowledge-Guide-Escalate) oriented and explicit guidelines and escalation logic to add reliability and clarity to the strategy. Results of this evaluation show that Phase 2 improved the Gemini 2.5 Flash model's classification accuracy to approximately 88%-89%, compared to 84%-86% in Phase 1, with empathy scores increasing from 4.2-4.3 to 4.4-4.8 on a five-point scale. The level of empathy also went up on a five-point scale to 4.4-4.8 as compared to 4.2-4.3. Discussion: The findings indicate that a structured, hierarchical approach improves the reliability and safety of psychosocial AI systems. The Acknowledge-Guide-Escalate (A.G.E.) framework enables consistent emotional recognition and contextually relevant guidance. The inclusion of human-in-the-loop oversight and preventive, actionable insights strengthens ethical alignment and response quality.