Beyond Words: Gaze-Enhanced LLM-based Dialogue System for Therapeutic Purposes

Mental health problems are becoming more common. Unfortunately, there are still not enough health specialists able to assist them. As a way to provide accessible and immediate support to those in need, digital interventions are receiving increased attention. Conversational agents based on large language models (LLMs) are increasingly used in mental health support. To help psychiatric patients, we have developed a therapeutic dialogue system called Terabot. Earlier experiments revealed several situations that paused the natural flow of the conversation with the patients. We created a new LLM-based dialogue system with an eye tracker as an additional input signal. We propose a feedback loop where the eye tracker gives the patient’s real-time gaze data into the dialogue system. Thanks to this, the patient can receive a response from Terabot, which is more suitable for the current dialogue situation. The dialogue agent’s responses can be improved, resulting in a more human-like flow of conversation.