Human-GenAI Comparison of a COSMIN Systematic Review: A Swiss Case Study
Abstract Background Artificial intelligence may accelerate systematic reviews of patient-reported outcome measures (PROMs), but its ability to identify eligible studies and reproduce Consensus-based Standards for the selection of health Measurement INstruments (COSMIN) assessments remains uncertain. Methods ChatGPT performance was compared with human reference decisions in a systematic review of generic PROMs relevant to Switzerland. Screening performance was evaluated across 4,374 title/abstract records and 1,512 full-text reports, of which 50 were ultimately included by human reviewers. Sensitivity, specificity, precision (positive predictive value), negative predictive value, and accuracy were calculated. A secondary, post-hoc analysis incorporated 12 human-verified corrections. Agreement on COSMIN measurement-property ratings, evidence quality, and overall recommendations was evaluated for five instruments. Results At title/abstract screening, ChatGPT correctly advanced 76 of 1,499 human-positive records (sensitivity 5.1%; precision 97.4%; specificity 99.9%). At full-text screening, ChatGPT correctly included 17 of 50 human-included studies and incorrectly included 11, yielding sensitivity of 34.0%, precision of 60.7%, and specificity of 99.2%. Following 12 human-verified corrections, sensitivity increased to 58.0% and precision to 72.5%. Agreement on matched COSMIN measurement-property ratings ranged from 52.9% (PROMIS) to 86.8% (SF-36), whereas evidence-quality agreement ranged from 34.6% (SF-36) to 62.5% (WORQ). Overall instrument recommendations agreed for four of five instruments (80%). Conclusions ChatGPT demonstrated low sensitivity for identifying eligible studies, despite high specificity. COSMIN agreement varied substantially, particularly for evidence-quality assessments. Human verification remains essential for reliable AI-assisted PROM systematic reviews.
Authors
- Thanh Elsener (ORCID: https://orcid.org/0009-0005-2888-3153)
- Matthew Kerry
Institutions
- ZHAW Zurich University of Applied Sciences (CH)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-13
- DOI
- https://doi.org/10.21203/rs.3.rs-11465622/v1
- Primary Topic
- Meta-analysis and systematic reviews
- Type
- preprint