A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study

Abstract Background Given that randomized controlled trials (RCTs) are the gold standard for effectiveness research, assessing their risk of bias (RoB) becomes an indispensable step in systematic reviews. Yet this task remains time-consuming and prone to reviewer subjectivity. While large language models may accelerate the process, they lack transparent, traceable judgments linked to trial reports. Objective To bridge methodological gaps and improve both efficiency and consistency, this study developed and externally validated TRoB-1, a fine-tuned open-weight large language model workflow for RoB assessment of RCTs. Methods TRoB-1 was developed specifically for the Cochrane risk of bias tool version 1 (RoB 1), which is the most widely adopted instrument for critical appraisal of RCTs. The workflow generates preliminary judgments and supporting quotations from reviewer-selected passages for the target RoB domains. The workflow was built on 2-stage supervised fine-tuning of Gemma 2-9B-Instruct using 4-bit quantized low-rank adaptation. The training set comprised 4321 RoB records extracted from systematic reviews of chronic obstructive pulmonary disease, diabetes, and hypertension. The validation set consisted of 161 records from recently published RCTs covering a broader disease area. The reference standard was established by 2 independent reviewers. Two clinicians blinded to the reference standard verified TRoB-1-assisted assessments through a locally hosted web interface. Using the same reference standard, the performance of TRoB-1 raw outputs was compared with that of the Gemma 2 base workflow, and clinician-verified outputs were compared with raw TRoB-1 outputs. Results TRoB-1 successfully generated structured outputs for all 161 validation records. In independent clinician-assisted assessments using the same workflow, raw TRoB-1 accuracy ranged from 73.3% (n=118) to 73.9% (n=119), compared with 52.2% (n=84) to 60.2% (n=97) for the Gemma 2 base model. After verification, accuracy ranged from 82.0% (n=132) to 84.5% (n=136) and macro- F 1 from 0.709 to 0.776. Accuracy remained higher for raw TRoB-1 after excluding invalid base-model outputs, and verification did not change any correct judgment to an incorrect one. Clinician 1 corrected 13 initially incorrect judgments and clinician 2 corrected 18. The F 1 -score for judgments of high-RoB increased from 0.600 to 0.784 for clinician 1 and from 0.571 to 0.821 for clinician 2. Conclusions TRoB-1 combined task-specific fine-tuning, evidence-linked outputs, and clinician verification in a transparent workflow for reviewer-supported RoB assessment. The TRoB-1 web interface provides a practical platform for integrating RoB into evidence synthesis, particularly for living systematic reviews where timely updates are critical.

Authors

Institutions

Publication Details

Journal
Journal of Medical Internet Research
Published
2026-11-13
DOI
https://doi.org/10.2196/108001
Primary Topic
Meta-analysis and systematic reviews
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study

Jinsong Geng, Guohua Wang, Xingyun Geng, Yitian Li et al.
6 links
Journal of Medical Internet Research
Meta-analysis and systematic reviews
article

A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study

Jinsong Geng, Guohua Wang, Xingyun Geng, Yitian Li, Jingyu Chen, Liting Liang
article en

Abstract

Abstract Background Given that randomized controlled trials (RCTs) are the gold standard for effectiveness research, assessing their risk of bias (RoB) becomes an indispensable step in systematic reviews. Yet this task remains time-consuming and prone to reviewer subjectivity. While large language models may accelerate the process, they lack transparent, traceable judgments linked to trial reports. Objective To bridge methodological gaps and improve both efficiency and consistency, this study developed and externally validated TRoB-1, a fine-tuned open-weight large language model workflow for RoB assessment of RCTs. Methods TRoB-1 was developed specifically for the Cochrane risk of bias tool version 1 (RoB 1), which is the most widely adopted instrument for critical appraisal of RCTs. The workflow generates preliminary judgments and supporting quotations from reviewer-selected passages for the target RoB domains. The workflow was built on 2-stage supervised fine-tuning of Gemma 2-9B-Instruct using 4-bit quantized low-rank adaptation. The training set comprised 4321 RoB records extracted from systematic reviews of chronic obstructive pulmonary disease, diabetes, and hypertension. The validation set consisted of 161 records from recently published RCTs covering a broader disease area. The reference standard was established by 2 independent reviewers. Two clinicians blinded to the reference standard verified TRoB-1-assisted assessments through a locally hosted web interface. Using the same reference standard, the performance of TRoB-1 raw outputs was compared with that of the Gemma 2 base workflow, and clinician-verified outputs were compared with raw TRoB-1 outputs. Results TRoB-1 successfully generated structured outputs for all 161 validation records. In independent clinician-assisted assessments using the same workflow, raw TRoB-1 accuracy ranged from 73.3% (n=118) to 73.9% (n=119), compared with 52.2% (n=84) to 60.2% (n=97) for the Gemma 2 base model. After verification, accuracy ranged from 82.0% (n=132) to 84.5% (n=136) and macro- F 1 from 0.709 to 0.776. Accuracy remained higher for raw TRoB-1 after excluding invalid base-model outputs, and verification did not change any correct judgment to an incorrect one. Clinician 1 corrected 13 initially incorrect judgments and clinician 2 corrected 18. The F 1 -score for judgments of high-RoB increased from 0.600 to 0.784 for clinician 1 and from 0.571 to 0.821 for clinician 2. Conclusions TRoB-1 combined task-specific fine-tuning, evidence-linked outputs, and clinician verification in a transparent workflow for reviewer-supported RoB assessment. The TRoB-1 web interface provides a practical platform for integrating RoB into evidence synthesis, particularly for living systematic reviews where timely updates are critical.

Journal of Medical Internet ResearchVol. 28
Capital Medical University (CN), Nantong University (CN)
Good health and well-being
Meta-analysis and systematic reviews
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon