A Comparative Analysis Between OpenEvidence and ChatGPT Using North American Spine Society Clinical Guidelines for Osteoporotic Vertebral Compression Fractures

Osteoporotic vertebral compression fractures are common fragility fractures associated with significant morbidity and mortality, and patients increasingly use artificial intelligence platforms to research treatment options. This study evaluated the guideline concordance of responses from ChatGPT 5.3, the model available to free users at the time of testing, compared with ChatGPT 5.5 with extended thinking and OpenEvidence using the 2024 North American Spine Society evidence-based guidelines. The 29 NASS clinical questions were individually submitted to OpenEvidence, ChatGPT 5.3, and ChatGPT 5.5 without prior chat history. Both ChatGPT models were accessed through the same paid ChatGPT Plus account. Two raters independently scored each response against all 74 NASS recommendations, and platforms were compared with paired (McNemar) tests. ChatGPT 5.5 was the most concordant, aligning with 53/74 recommendations (71.6%) and 34/37 evidence-supported grade A/B/C recommendations (91.9%). OpenEvidence was concordant with 35/74 recommendations (47.3%) and 31/37 A/B/C recommendations (83.8%). ChatGPT 5.3 was concordant with 24/74 recommendations (32.4%) and 22/37 A/B/C recommendations (59.5%). ChatGPT 5.3 showed substantially lower concordance than OpenEvidence and ChatGPT 5.5 and was the only chatbot to directly contradict NASS recommendations; it cited sources in only 15 of 29 responses. For all platforms, the most frequent errors were omission of recommendations and overstatement of the strength of evidence. In the configurations tested, the model available to free users was less concordant than the paid reasoning configuration and the medically focused platform; because the platforms differ in model generation, reasoning mode, access tier, and retrieval, the source of this difference cannot be isolated. Patients should use AI only as an adjunct to physician-guided care.

Authors

Institutions

Publication Details

Journal
AI in Medicine
Published
2026-11-13
DOI
https://doi.org/10.3390/aimed1040030
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Comparative Analysis Between OpenEvidence and ChatGPT Using North American Spine Society Clinical Guidelines for Osteoporotic Vertebral Compression Fractures

Theresa J. C. Pazionis, Funiba Ngwa, Sebastian Kiehn
7 links
AI in Medicine
Artificial Intelligence in Healthcare and Education
article

A Comparative Analysis Between OpenEvidence and ChatGPT Using North American Spine Society Clinical Guidelines for Osteoporotic Vertebral Compression Fractures

Theresa J. C. Pazionis, Funiba Ngwa, Sebastian Kiehn
article en

Abstract

Osteoporotic vertebral compression fractures are common fragility fractures associated with significant morbidity and mortality, and patients increasingly use artificial intelligence platforms to research treatment options. This study evaluated the guideline concordance of responses from ChatGPT 5.3, the model available to free users at the time of testing, compared with ChatGPT 5.5 with extended thinking and OpenEvidence using the 2024 North American Spine Society evidence-based guidelines. The 29 NASS clinical questions were individually submitted to OpenEvidence, ChatGPT 5.3, and ChatGPT 5.5 without prior chat history. Both ChatGPT models were accessed through the same paid ChatGPT Plus account. Two raters independently scored each response against all 74 NASS recommendations, and platforms were compared with paired (McNemar) tests. ChatGPT 5.5 was the most concordant, aligning with 53/74 recommendations (71.6%) and 34/37 evidence-supported grade A/B/C recommendations (91.9%). OpenEvidence was concordant with 35/74 recommendations (47.3%) and 31/37 A/B/C recommendations (83.8%). ChatGPT 5.3 was concordant with 24/74 recommendations (32.4%) and 22/37 A/B/C recommendations (59.5%). ChatGPT 5.3 showed substantially lower concordance than OpenEvidence and ChatGPT 5.5 and was the only chatbot to directly contradict NASS recommendations; it cited sources in only 15 of 29 responses. For all platforms, the most frequent errors were omission of recommendations and overstatement of the strength of evidence. In the configurations tested, the model available to free users was less concordant than the paid reasoning configuration and the medically focused platform; because the platforms differ in model generation, reasoning mode, access tier, and retrieval, the source of this difference cannot be isolated. Patients should use AI only as an adjunct to physician-guided care.

AI in MedicineVol. 1(4)
Temple University Hospital (US), Temple University (US)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon