RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine developed from a scoping review

Large language models (LLMs) are increasingly used to evaluate other models’ outputs in medicine, an approach known as LLM-as-a-Judge (LaaJ), yet no LaaJ-specific reporting checklist exists. This scoping review included 91 medical LaaJ studies: 69 from 5,571 database records and 22 from a previous review. Judges were mainly used for response evaluation (74.7%), and 76.9% of the studies used at least one proprietary LLM. Agreement with human evaluators was reported in 65.9% of the studies, with widely varying metrics. Based on these findings, this study developed RULE-LLM, a reporting framework of 21 items and eight sub-items across four domains: role and task specification, underlying data and criteria, LLM-Judge design and output, and evaluation of judge performance. Seven experts rated the initial draft (S-CVI/Ave, 0.94), and five independent raters showed substantial agreement when applying the final framework (Fleiss' κ, 0.67; Gwet's AC1, 0.73). RULE-LLM aims to improve the transparency and reproducibility of LaaJ studies.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-19
DOI
https://doi.org/10.21203/rs.3.rs-11611263/v1
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine developed from a scoping review

Yoon Bin Jung, Junbok Lee, Huiyeol Yun
Research Square
Artificial Intelligence in Healthcare and Education
preprint

RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine developed from a scoping review

Yoon Bin Jung, Junbok Lee, Huiyeol Yun
preprint en

Abstract

Large language models (LLMs) are increasingly used to evaluate other models’ outputs in medicine, an approach known as LLM-as-a-Judge (LaaJ), yet no LaaJ-specific reporting checklist exists. This scoping review included 91 medical LaaJ studies: 69 from 5,571 database records and 22 from a previous review. Judges were mainly used for response evaluation (74.7%), and 76.9% of the studies used at least one proprietary LLM. Agreement with human evaluators was reported in 65.9% of the studies, with widely varying metrics. Based on these findings, this study developed RULE-LLM, a reporting framework of 21 items and eight sub-items across four domains: role and task specification, underlying data and criteria, LLM-Judge design and output, and evaluation of judge performance. Seven experts rated the initial draft (S-CVI/Ave, 0.94), and five independent raters showed substantial agreement when applying the final framework (Fleiss' κ, 0.67; Gwet's AC1, 0.73). RULE-LLM aims to improve the transparency and reproducibility of LaaJ studies.

Research Square
Yonsei University (KR)
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon