RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine developed from a scoping review
Large language models (LLMs) are increasingly used to evaluate other models’ outputs in medicine, an approach known as LLM-as-a-Judge (LaaJ), yet no LaaJ-specific reporting checklist exists. This scoping review included 91 medical LaaJ studies: 69 from 5,571 database records and 22 from a previous review. Judges were mainly used for response evaluation (74.7%), and 76.9% of the studies used at least one proprietary LLM. Agreement with human evaluators was reported in 65.9% of the studies, with widely varying metrics. Based on these findings, this study developed RULE-LLM, a reporting framework of 21 items and eight sub-items across four domains: role and task specification, underlying data and criteria, LLM-Judge design and output, and evaluation of judge performance. Seven experts rated the initial draft (S-CVI/Ave, 0.94), and five independent raters showed substantial agreement when applying the final framework (Fleiss' κ, 0.67; Gwet's AC1, 0.73). RULE-LLM aims to improve the transparency and reproducibility of LaaJ studies.
Authors
- Yoon Bin Jung (ORCID: https://orcid.org/0000-0001-9829-1931)
- Junbok Lee (ORCID: https://orcid.org/0000-0001-7770-4362)
- Huiyeol Yun
Institutions
- Yonsei University (KR)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-19
- DOI
- https://doi.org/10.21203/rs.3.rs-11611263/v1
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- preprint