2054 Comparison of Quality, Empathy and Readability of Physician Responses Versus Chatbot Responses to Common Vascular Neurosurgery-Related Questions on a Social Media Platform

INTRODUCTION: Social media platforms are utilized by patients prior to scheduling formal consultations. They also serve as a means of pursuing second opinions. Vascular pathologies require regular surveillance and specialized care. In recent years, chatbots have been trained using large language models (LLMs) to provide information on neurosurgical conditions. However, their ability to answer vascular neurosurgery-related questions has not been compared to physician responses on social media. METHODS: We identified the top 50 vascular neurosurgery-related questions and subsequent verified physician responses from the Reddit social media platform. These responses were inputted into ChatGPT. Responses were anonymized and ranked on a Likert scale for accuracy regarding neurosurgical guidelines, completeness (ability to address all aspects of the question) and empathy by four independent reviewers. Readability was assessed using standardized indexes (Flesch Reading Ease, Flesch Kincaid Grade, Gunning Fox Index, SMOG Index, Automated Readability Index and Coleman Liau Index). Mean Likert scale ratings and readability were compared using 2-tailed t-tests. RESULTS: Responses provided by ChatGPT were perceived to have significantly higher ratings of completeness (mean ± SD) 2.62 ± 0.64 and empathy 3.96 ± 1.08, compared to physician responses (1.96 ± 0.75 and 2.22 ± 1.02), respectively (P<0.001). Accuracy of healthcare information did not differ significantly. In terms of readability, physician responses had significantly higher ease of readability and lower grade-level readability compared to ChatGPT (P<0.001). CONCLUSIONS: Our results suggest higher empathy and completeness of information provided by ChatGPT compared to physicians. However, these responses are at readability levels higher than the literacy of the average American population (sixth-grade readability). Future research could emphasize incorporating chatbot responses while drafting physician responses to provide more balanced information to healthcare questions.