A Comparative Analysis Between OpenEvidence and ChatGPT Using North American Spine Society Clinical Guidelines for Osteoporotic Vertebral Compression Fractures
Osteoporotic vertebral compression fractures are common fragility fractures associated with significant morbidity and mortality, and patients increasingly use artificial intelligence platforms to research treatment options. This study evaluated the guideline concordance of responses from ChatGPT 5.3, the model available to free users at the time of testing, compared with ChatGPT 5.5 with extended thinking and OpenEvidence using the 2024 North American Spine Society evidence-based guidelines. The 29 NASS clinical questions were individually submitted to OpenEvidence, ChatGPT 5.3, and ChatGPT 5.5 without prior chat history. Both ChatGPT models were accessed through the same paid ChatGPT Plus account. Two raters independently scored each response against all 74 NASS recommendations, and platforms were compared with paired (McNemar) tests. ChatGPT 5.5 was the most concordant, aligning with 53/74 recommendations (71.6%) and 34/37 evidence-supported grade A/B/C recommendations (91.9%). OpenEvidence was concordant with 35/74 recommendations (47.3%) and 31/37 A/B/C recommendations (83.8%). ChatGPT 5.3 was concordant with 24/74 recommendations (32.4%) and 22/37 A/B/C recommendations (59.5%). ChatGPT 5.3 showed substantially lower concordance than OpenEvidence and ChatGPT 5.5 and was the only chatbot to directly contradict NASS recommendations; it cited sources in only 15 of 29 responses. For all platforms, the most frequent errors were omission of recommendations and overstatement of the strength of evidence. In the configurations tested, the model available to free users was less concordant than the paid reasoning configuration and the medically focused platform; because the platforms differ in model generation, reasoning mode, access tier, and retrieval, the source of this difference cannot be isolated. Patients should use AI only as an adjunct to physician-guided care.
Authors
- Theresa J. C. Pazionis (ORCID: https://orcid.org/0000-0003-3894-3605)
- Funiba Ngwa (ORCID: https://orcid.org/0009-0000-6860-2233)
- Sebastian Kiehn
Institutions
- Temple University Hospital (US)
- Temple University (US)
Publication Details
- Journal
- AI in Medicine
- Published
- 2026-11-13
- DOI
- https://doi.org/10.3390/aimed1040030
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00