Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers
ChatGPT appears to have some ability to assess RoB and is unlikely to be guessing or "hallucinating". The estimated agreement for "Overall RoB" is well above estimates of agreement reported for some human reviewers, but below the highest estimates. LLM-based systems for assessing RoB may be able to help streamline and improve evidence synthesis production.
Authors
- Ashley Elizabeth Muller (ORCID: https://orcid.org/0000-0001-7819-6697)
- Julie Glanville (ORCID: https://orcid.org/0000-0002-1253-8524)
- Rigmor C. Berg (ORCID: https://orcid.org/0000-0002-6915-0993)
- Christopher James Rose (ORCID: https://orcid.org/0000-0001-6457-8168)
- Julia Bidonde (ORCID: https://orcid.org/0000-0001-7535-678X)
- Hans Bugge Bergsund (ORCID: https://orcid.org/0000-0003-4902-0507)
- Martin Ringsten (ORCID: https://orcid.org/0000-0003-4456-9760)
- Chris Cooper (ORCID: https://orcid.org/0000-0003-0864-5607)
- José F. Meneses-Echávez (ORCID: https://orcid.org/0000-0003-4312-6909)
- Thomas Potrebny (ORCID: https://orcid.org/0000-0003-1487-0560)
Institutions
- Norwegian Institute of Public Health (NO)
- Western Norway University of Applied Sciences (NO)
- Universidad Santo Tomás (CO)
- University of Saskatchewan (CA)
- University of Bristol (GB)
- Centre for Arctic Gas Hydrate, Environment and Climate (NO)
- Skåne University Hospital (SE)
- Western Norway University of Applied Sciences (NO)
- University of York (GB)
- UiT The Arctic University of Norway (NO)
Publication Details
- Journal
- Cochrane Evidence Synthesis and Methods
- Published
- 2025-09-01
- DOI
- https://doi.org/10.1002/cesm.70048
- Citations
- 1
- Primary Topic
- Meta-analysis and systematic reviews
- Type
- article
- Field-Weighted Citation Impact
- 1.40
Funders
- Norwegian Institute of Public Health