Building Conversational Agents for Stroke Rehabilitation: An Evaluation of Large Language Models and Retrieval Augmented Generation

Conversational agents (CAs) have been successfully implemented to deliver digital health interventions. However, most of their applications focus on non-communicable diseases, e.g., diabetes. RehabCoach, a CA-based smartphone application designed to assist stroke survivors during at-home unsupervised rehabilitation, utilizes multiple CAs to deliver digital interventions for therapy adherence. Matthias, one of the CAs of RehabCoach, is a novel Large Language Model-based CA that leverages Retrieval Augmented Generation (RAG) to answer questions related to ReHandyBot, an upper limb rehabilitation device. In this work, we assessed the preciseness and conciseness of Matthias’s answers to 14 device-related questions. For each question, Matthias generated 6 answers, based on 3 different prompts and 2 different data sources, i.e., a conversational dataset containing dialogues about how to operate the device and the device manual of instructions. Results from a blinded evaluation performed by two device experts highlight that the responses of the CA generated using the conversational dataset were considered more concise (82,14%) and better overall (60,70%). This work shows the potential of using conversational data for device-assisted rehabilitation CAs deploying RAG.