Human evaluation of large language models in healthcare: gaps, challenges, and the need for standardization

Publications related to experimentation with Large Language Models (LLMs) in healthcare are rapidly increasing. While human evaluation remains the gold standard for evaluating LLMs, there is still a lack of standardization in its implementation. In this review article, we systematically examine studies involving LLMs in healthcare that have conducted human evaluations. We analyze the metrics used, assess their variability across studies. We also propose a standardized framework along with an interactive open web application HumanELY, to facilitate human evaluation. We believe that use of HumanELY will provide an opportunity for consistent, comprehensive, reliable, reproducible, and measurable human evaluations of LLM in healthcare. HumanELY is publicly available at https://www.brainxai.com/humanely .

Authors

Institutions

Publication Details

Journal
npj Health Systems
Published
2025-11-03
DOI
https://doi.org/10.1038/s44401-025-00043-2
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Human evaluation of large language models in healthcare: gaps, challenges, and the need for standardization

Jacek B. Cywiński, S. Ramachandran, Shreya Mishra, Anabelle Cohen et al.
npj Health Systems
Artificial Intelligence in Healthcare and Education
article

Human evaluation of large language models in healthcare: gaps, challenges, and the need for standardization

Jacek B. Cywiński, S. Ramachandran, Shreya Mishra, Anabelle Cohen, Raghav Awasthi, Francis Papay, Kamal Maheshwari, Piyush Mathur, Ashish Atreja, Rahul Kashyap, Hajra Arshad, Dwarikanath Mahapatra, Nishant Kumar Singh, Ashish K. Khanna, Izabella DiRosa, Atharva Bhattad, Atulya Atreja, Asma Alshukaili, Aryan Vohra
article en

Abstract

Publications related to experimentation with Large Language Models (LLMs) in healthcare are rapidly increasing. While human evaluation remains the gold standard for evaluating LLMs, there is still a lack of standardization in its implementation. In this review article, we systematically examine studies involving LLMs in healthcare that have conducted human evaluations. We analyze the metrics used, assess their variability across studies. We also propose a standardized framework along with an interactive open web application HumanELY, to facilitate human evaluation. We believe that use of HumanELY will provide an opportunity for consistent, comprehensive, reliable, reproducible, and measurable human evaluations of LLM in healthcare. HumanELY is publicly available at https://www.brainxai.com/humanely .

npj Health SystemsVol. 2(1)
Cleveland Clinic (US), WellSpan Health (US), BrainAid (United States) (US), Case Western Reserve University (US)
Openalex Percentile: Top 75%
Artificial Intelligence in Healthcare and Education
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.