Validation of a large language model–assisted three-layer screening cascade against a completed clinical practice guideline for hip osteoarthritis

ABSTRACT Purpose Clinical practice guideline development requires title and abstract screening of many records. Large language models (LLMs) may reduce the number of records requiring manual screening, but their ability to retain studies identified during full-text assessment remains uncertain. Methods We retrospectively evaluated an LLM-assisted three-layer screening cascade using 904 English-language records from chapter 4 (conservative management) of a completed clinical practice guideline for hip osteoarthritis. The cascade used GPT-4o mini to assess study design, population, and question-specific intervention and outcome criteria. After full-text assessment, the guideline panel selected 64 record-question pairs, which served as the reference standard. Uncertain records were retained for manual review. The full cascade was run independently five times. The primary outcomes were the reduction in records requiring manual screening and sensitivity against the reference standard. Results Across the five runs, a median of 420 of 904 records (46.5%) were automatically excluded from all eight questions, leaving a median of 484 records (53.5%) for review on at least one question. For individual questions, 40–272 records remained for manual screening. The cascade retained 60 of 64 reference standard record-question pairs in each run, yielding a sensitivity of 93.8% (95% confidence interval, 85.0%–97.5%). The four pairs not retained comprised two safety studies, one biomechanical study, and one systematic review mainly addressing knee osteoarthritis. Conclusions In chapter 4 of a completed hip osteoarthritis guideline, an LLM-assisted cascade reduced the number of records requiring manual title and abstract screening while retaining 60 of 64 selected pairs. Its applicability to other chapters and guidelines, and to evidence beyond direct intervention effectiveness studies, remains to be evaluated.