A Randomized Trial Assessing the Impact of an Artificial Intelligence Chatbot on Patient Decision-Making

INTRODUCTION: Artificial Intelligence (AI) platforms such as large language models (LLMs) increasingly have been incorporated into patient-facing healthcare. Although LLMs specific to urogynecology are under development as decisional aids, there has been no study reporting how LLMs that are generally available to patients (e.g., ChatGPT) may affect their decisional conflict. OBJECTIVE: To investigate the effect of use of an artificial intelligence chatbot platform on patient decisional conflict at the time of their initial urogynecology visit. METHODS: This IRB-approved, single-center randomized study examined patient use of an LLM, ChatGPT-4o (OpenAI), during the initial visit to aid in the decision-making regarding their urogynecologic complaint(s). Participants were recruited at their first urogynecology visit if they presented with prolapse, incontinence, or lower urinary tract symptoms (LUTS) and were English- or Spanish-speaking. They were randomized into one of three arms: use of ChatGPT prior to their appointment, use of ChatGPT following their appointment, or no use of an LLM (usual care). Based on their study arm, patients were provided ChatGPT on a tablet, and instructed to ask (typing or speaking) the program anything they’d like about their most important urogynecology problem, with up to five follow-up questions. Demographic data, previous methods of obtaining medical information, and visit details were collected. Patients were asked to complete surveys immediately and 3 months following their visit evaluating their understanding of the diagnosis and treatment plans, decisional conflict (Decisional Conflict Scale), visit satisfaction, and chatbot satisfaction. Significant findings were declared at p<0.05. RESULTS: 125 patients were randomized from July to December, 2024, with a 3.2% rate of attrition at the 3-month survey. The majority of participants identified as White (78.2%), with at least a high-school education (92.8%). 8% of patients were Spanish-speaking. Majority (98.8%) reported no prior chatbot use in their medical care. Other than age, there was no difference in demographics and symptom scores (Table 1). Primary diagnoses of prolapse (30.4%), incontinence (48.8%), and LUTS (20.0%) did not differ between groups. Decisional conflict was low, and there was no difference between groups immediately (p=0.691) nor at 3 (p=0.875) months after the visit (Figure 1). Participant responses regarding understanding of diagnosis, visit satisfaction, and chatbot satisfaction did not differ by study arm immediately or at 3 months (Table 2). A greater number of patients in the pre- and post-visit chatbot use groups reported home use of a chatbot by 3 months compared with the no chatbot group (pre-visit 33.3%, post-visit 13.2%, no chatbot 7.9%; p=0.011). There was no significant correlation between doctor-reported patient understanding of diagnosis and patient-reported understanding immediately post-visit (R=0.137, p=0.130), but a significant correlation at 3 months (R=0.183, p=0.045). CONCLUSIONS: Patient use of an AI chatbot peri-visit did not affect decisional conflict at the initial urogynecologic visit. While AI chatbots are increasingly accepted as an adjunct for patients investigating their conditions, they may not yet serve as the optimal decision-making tool for information gathering or supporting patient-centered goal attainment.Figure 1Table 1Table 2