1133 Evaluating the Accuracy, Variation, And Sources of Large Language Model Responses to Patient Questions in Neurosurgery: An Assessment of Popular Large Language Models and Emerging Platforms

INTRODUCTION: Artificial intelligence (AI) has the potential to provide accurate information and improve the accessibility of medical literature. With the use of AI growing, a neurosurgery specific platform, AtlasGPT, has been developed. METHODS: Popular public LLMs, such as ChatGPT, Perplexity.AI, and BingAI, were assessed, as well as AtlasGPTs in regard to neurosurgical patient focused questions. LLM responses were assessed for accuracy, response variation, and the references used in their responses were recorded. ChatGPT-3.5 was omitted from the reference analysis due to known fabrication of references. RESULTS: Across 120 responses from each platform, there was no significant difference in response accuracy between platforms, though AtlasGPT answered 100% of responses correctly, while publicly available platforms ranged from 80-87%. In regard to accuracy variation, AtlasGPT (0%) had significantly less variation than ChatGPT3.5 and 4.0 (p<0.001). Reference analysis showed that popular public LLMs tended to mostly reference peer reviewed articles, but also cited medical device companies, news outlets and blogs. ChatGPT-4.0 was unable to provide a reference in 36% of responses. CONCLUSIONS: Developments in AI tools for information for patients in neurosurgery and medicine as a whole should focus on creating resources with carefully curated information sources and consistent, accurate responses for patients. Increasing the accessibility of patients to accurate answers in regard to patient questions has the potential to decrease information asymmetry and improve outcomes.