Guideline-Concordant Answer Quality of Five Large Language Models on Indication Criteria for Total Hip Arthroplasty in Hip Osteoarthritis: A Single-Run Benchmark Against the 2021 German S3 Guideline (EKIT-Hip)

Background/Objectives: Hip osteoarthritis (OA) is a highly prevalent musculoskeletal disorder associated with substantial functional impairment and healthcare burden. Although evidence-based clinical practice guidelines exist, including the 2021 guideline of the German Society for Orthopaedics and Trauma Surgery (DGOU), their implementation in routine practice remains incomplete. Large language models (LLMs) are increasingly consulted for guideline-related information, but the concordance of their answers with guideline recommendations is uncertain. This benchmark study evaluated the guideline concordance, comprehensiveness and length of answers generated by five widely used LLMs to questions derived from the 2021 German S3 guideline on evidence- and consensus-based indication criteria for total hip arthroplasty in hip OA (EKIT-Hip; AWMF register no. 187-001; lead society DGOU). It did not evaluate the effectiveness or safety of clinical decision support. Methods: In a cross-sectional, single-run benchmark study, 26 structured, context-free questions covering five of the six thematic complexes of the guideline (diagnosis and imaging; assessment of the level of suffering; evaluation of alternative therapeutic options; contraindications; optimisation of modifiable risk factors) were derived from the guideline recommendations and submitted between 9 November 2024 and 25 February 2025 through the consumer web interfaces of five LLM-based systems (GPT-4, GPT-4o, GPT-4o mini, Gemini Advanced and Consensus Premium), yielding 130 responses (one per system–question pair, each in a newly initiated chat session). Responses were evaluated by three board-certified orthopaedic surgeons blinded to model identity, using a four-point accuracy scale and a five-point comprehensiveness scale. Only a single consensus score per response was recorded, so inter-rater reliability could not be assessed. Response length was quantified in words and characters and benchmarked against the corresponding guideline text. Because all systems answered the same questions, ratings were analysed as paired, question-clustered data with cumulative-link mixed models (random intercept for question), Friedman and Cochran’s Q tests and question-level cluster-bootstrap confidence intervals; unpaired analyses were retained as sensitivity analyses. Results: Accuracy differed across systems (cumulative-link mixed model, likelihood-ratio χ2(4) = 49.0, p < 0.0001; paired Friedman χ2(4) = 41.7, p < 0.0001; unpaired Kruskal–Wallis sensitivity analysis H = 35.6, p < 0.0001). Mean accuracy was highest for GPT-4o (3.69 ± 0.55) and GPT-4 (3.62 ± 0.57), followed by GPT-4o mini (3.27 ± 0.78), Gemini Advanced (3.08 ± 0.84) and Consensus Premium (2.54 ± 0.71). Responses rated good or excellent accounted for 25/26 (96.2%) of GPT-4o and GPT-4 responses versus 15/26 (57.7%) for Consensus Premium. Comprehensiveness followed the same ordering (H = 33.0, p < 0.0001, ε2 = 0.23), with GPT-4o achieving the highest mean score (4.31 ± 0.79). In the cumulative-link mixed model, the odds of a higher accuracy rating relative to Consensus Premium were 51.8 (95% CI 12.4–215.9) for GPT-4o and 35.4 (95% CI 9.2–136.4) for GPT-4; GPT-4o and GPT-4 did not differ (OR 1.47, multiplicity-adjusted 95% CI 0.23–9.21), and their paired mean difference in accuracy (0.08 points, 90% CI −0.11 to 0.26) lay within an equivalence margin of ±0.5 points. Responses rated poor were observed for Consensus Premium (2/26) and Gemini Advanced (1/26) and not for the ChatGPT variants (0/26 each; exact 95% CI 0–13.2%), so a clinically relevant error rate cannot be excluded for any system. Response length differed markedly (Friedman χ2 = 47.3, p < 0.0001), ranging from 32.4 ± 11.5 words for Consensus Premium to 77.8 ± 24.9 words for GPT-4o, corresponding to 1.29-fold and 3.19-fold the length of the source guideline text, respectively. Conclusions: GPT-4o and GPT-4 received the highest ratings for accuracy and comprehensiveness in this single-run benchmark of static, context-free questions. These data describe guideline-concordant answer quality only; they do not establish clinical decision-support effectiveness or safety and do not support unsupervised clinical use of any evaluated system. Evaluation with standardised patient cases, repeated version-controlled runs, clinician-in-the-loop designs and prospective safety endpoints is required.