Human-GenAI Comparison of a COSMIN Systematic Review: A Swiss Case Study

Abstract Background Artificial intelligence may accelerate systematic reviews of patient-reported outcome measures (PROMs), but its ability to identify eligible studies and reproduce Consensus-based Standards for the selection of health Measurement INstruments (COSMIN) assessments remains uncertain. Methods ChatGPT performance was compared with human reference decisions in a systematic review of generic PROMs relevant to Switzerland. Screening performance was evaluated across 4,374 title/abstract records and 1,512 full-text reports, of which 50 were ultimately included by human reviewers. Sensitivity, specificity, precision (positive predictive value), negative predictive value, and accuracy were calculated. A secondary, post-hoc analysis incorporated 12 human-verified corrections. Agreement on COSMIN measurement-property ratings, evidence quality, and overall recommendations was evaluated for five instruments. Results At title/abstract screening, ChatGPT correctly advanced 76 of 1,499 human-positive records (sensitivity 5.1%; precision 97.4%; specificity 99.9%). At full-text screening, ChatGPT correctly included 17 of 50 human-included studies and incorrectly included 11, yielding sensitivity of 34.0%, precision of 60.7%, and specificity of 99.2%. Following 12 human-verified corrections, sensitivity increased to 58.0% and precision to 72.5%. Agreement on matched COSMIN measurement-property ratings ranged from 52.9% (PROMIS) to 86.8% (SF-36), whereas evidence-quality agreement ranged from 34.6% (SF-36) to 62.5% (WORQ). Overall instrument recommendations agreed for four of five instruments (80%). Conclusions ChatGPT demonstrated low sensitivity for identifying eligible studies, despite high specificity. COSMIN agreement varied substantially, particularly for evidence-quality assessments. Human verification remains essential for reliable AI-assisted PROM systematic reviews.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11465622/v1
Primary Topic
Meta-analysis and systematic reviews
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Human-GenAI Comparison of a COSMIN Systematic Review: A Swiss Case Study

Thanh Elsener, Matthew Kerry
3 links
Research Square
Meta-analysis and systematic reviews
preprint

Human-GenAI Comparison of a COSMIN Systematic Review: A Swiss Case Study

Thanh Elsener, Matthew Kerry
preprint en

Abstract

Abstract Background Artificial intelligence may accelerate systematic reviews of patient-reported outcome measures (PROMs), but its ability to identify eligible studies and reproduce Consensus-based Standards for the selection of health Measurement INstruments (COSMIN) assessments remains uncertain. Methods ChatGPT performance was compared with human reference decisions in a systematic review of generic PROMs relevant to Switzerland. Screening performance was evaluated across 4,374 title/abstract records and 1,512 full-text reports, of which 50 were ultimately included by human reviewers. Sensitivity, specificity, precision (positive predictive value), negative predictive value, and accuracy were calculated. A secondary, post-hoc analysis incorporated 12 human-verified corrections. Agreement on COSMIN measurement-property ratings, evidence quality, and overall recommendations was evaluated for five instruments. Results At title/abstract screening, ChatGPT correctly advanced 76 of 1,499 human-positive records (sensitivity 5.1%; precision 97.4%; specificity 99.9%). At full-text screening, ChatGPT correctly included 17 of 50 human-included studies and incorrectly included 11, yielding sensitivity of 34.0%, precision of 60.7%, and specificity of 99.2%. Following 12 human-verified corrections, sensitivity increased to 58.0% and precision to 72.5%. Agreement on matched COSMIN measurement-property ratings ranged from 52.9% (PROMIS) to 86.8% (SF-36), whereas evidence-quality agreement ranged from 34.6% (SF-36) to 62.5% (WORQ). Overall instrument recommendations agreed for four of five instruments (80%). Conclusions ChatGPT demonstrated low sensitivity for identifying eligible studies, despite high specificity. COSMIN agreement varied substantially, particularly for evidence-quality assessments. Human verification remains essential for reliable AI-assisted PROM systematic reviews.

Research Square
ZHAW Zurich University of Applied Sciences (CH)
Good health and well-being
Meta-analysis and systematic reviews
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon