Concordance of GPT-5.6, Gemini 3.6 Flash, DeepSeek-V4, and Doubao-2.1 With the Formal EU-US Meniscus Rehabilitation 2024 Consensus: A Comparative Study at the Question and Statement Levels

Background Large language models (LLMs) are increasingly used for clinical information retrieval in orthopaedics and sports medicine, but their value depends on fidelity to the relevant guideline or consensus. This study compared four LLMs with the formal 2024 ESSKA-AOSSM-AASPT EU-US Meniscus Rehabilitation Consensus in a realistic setting in which the consensus was named but its source text was not provided. Methods This consensus-referenced cross-sectional comparative study evaluated GPT-5.6, Gemini 3.6 Flash, DeepSeek-V4, and Doubao-2.1 using 37 clinical questions and 52 disaggregated recommendation statements. All models received identical instructions to answer strictly according to the consensus, without access to its PDF, full text, recommendation list, reference answers, or scoring criteria. Outcomes included question-level accuracy, strict statement-level concordance, at-least-partial coverage, error types, and potentially hazardous omissions. Paired differences were analyzed using Cochran's Q test and exact McNemar tests with Holm correction. Results Question-level accuracy differed among models (Cochran's Q = 9.09, p = 0.028): DeepSeek-V4 achieved 25/37 (67.6%), GPT-5.6 24/37 (64.9%), Gemini 3.6 Flash 23/37 (62.2%), and Doubao-2.1 16/37 (43.2%); no pairwise comparison remained significant after Holm correction. Strict statement-level concordance differed more markedly (Cochran's Q = 27.53, p < 0.001): GPT-5.6 achieved 25/52 (48.1%), Gemini 3.6 Flash 22/52 (42.3%), DeepSeek-V4 10/52 (19.2%), and Doubao-2.1 7/52 (13.5%). Errors primarily involved weight-bearing and range-of-motion restrictions, procedure-specific rehabilitation, return-to-sport criteria, outcome measures, and referral criteria. Conclusions LLM performance was heterogeneous when the consensus was named but not supplied. Question-level correctness did not ensure complete coverage of procedure-specific restrictions. LLMs may provide educational support, but individualized rehabilitation decisions require verification against the original consensus and clinical instructions.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-18
DOI
https://doi.org/10.21203/rs.3.rs-11193068/v1
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Concordance of GPT-5.6, Gemini 3.6 Flash, DeepSeek-V4, and Doubao-2.1 With the Formal EU-US Meniscus Rehabilitation 2024 Consensus: A Comparative Study at the Question and Statement Levels

Xu Liang, Qichun Zhao, Yingming Wang, Dongjing Fang et al.
7 links
Research Square
Artificial Intelligence in Healthcare and Education
preprint

Concordance of GPT-5.6, Gemini 3.6 Flash, DeepSeek-V4, and Doubao-2.1 With the Formal EU-US Meniscus Rehabilitation 2024 Consensus: A Comparative Study at the Question and Statement Levels

Xu Liang, Qichun Zhao, Yingming Wang, Dongjing Fang, Xiaohe Wang
preprint en

Abstract

Background Large language models (LLMs) are increasingly used for clinical information retrieval in orthopaedics and sports medicine, but their value depends on fidelity to the relevant guideline or consensus. This study compared four LLMs with the formal 2024 ESSKA-AOSSM-AASPT EU-US Meniscus Rehabilitation Consensus in a realistic setting in which the consensus was named but its source text was not provided. Methods This consensus-referenced cross-sectional comparative study evaluated GPT-5.6, Gemini 3.6 Flash, DeepSeek-V4, and Doubao-2.1 using 37 clinical questions and 52 disaggregated recommendation statements. All models received identical instructions to answer strictly according to the consensus, without access to its PDF, full text, recommendation list, reference answers, or scoring criteria. Outcomes included question-level accuracy, strict statement-level concordance, at-least-partial coverage, error types, and potentially hazardous omissions. Paired differences were analyzed using Cochran's Q test and exact McNemar tests with Holm correction. Results Question-level accuracy differed among models (Cochran's Q = 9.09, p = 0.028): DeepSeek-V4 achieved 25/37 (67.6%), GPT-5.6 24/37 (64.9%), Gemini 3.6 Flash 23/37 (62.2%), and Doubao-2.1 16/37 (43.2%); no pairwise comparison remained significant after Holm correction. Strict statement-level concordance differed more markedly (Cochran's Q = 27.53, p < 0.001): GPT-5.6 achieved 25/52 (48.1%), Gemini 3.6 Flash 22/52 (42.3%), DeepSeek-V4 10/52 (19.2%), and Doubao-2.1 7/52 (13.5%). Errors primarily involved weight-bearing and range-of-motion restrictions, procedure-specific rehabilitation, return-to-sport criteria, outcome measures, and referral criteria. Conclusions LLM performance was heterogeneous when the consensus was named but not supplied. Question-level correctness did not ensure complete coverage of procedure-specific restrictions. LLMs may provide educational support, but individualized rehabilitation decisions require verification against the original consensus and clinical instructions.

Research Square
University of Science and Technology of China (CN)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon