MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models
Ensuring the general efficacy and benefit for human beings from medical Large Language Models (LLM) before real-world deployment is crucial. However, a widely accepted and accessible evaluation process for medical LLM, especially in the Chinese context, remains to be established. In this work, we introduce “MedBench”, a comprehensive, standardized, and reliable benchmarking system for Chinese medical LLM. First, MedBench assembles the currently largest evaluation dataset (300 901 questions) to cover 43 clinical specialties, and performs multi-faceted evaluation on medical LLM. Second, MedBench provides a standardized and fully automatic cloud-based evaluation infrastructure, with physical separations between question and ground truth. Third, MedBench implements dynamic evaluation mechanisms to prevent shortcut learning and answer memorization. Applying MedBench to popular general and medical LLMs, we observe unbiased, reproducible evaluation results largely aligning with medical professionals' perspectives. This study establishes a significant foundation for preparing the practical applications of Chinese medical LLMs. MedBench is publicly accessible at https://medbench.opencompass.org.cn.
Authors
- Tong Ruan (ORCID: https://orcid.org/0000-0002-3546-8338)
- Xiaoming Shi (ORCID: https://orcid.org/0000-0002-7071-571X)
- Weiguo Hu (ORCID: https://orcid.org/0000-0001-7420-6640)
- Benyou Wang (ORCID: https://orcid.org/0000-0002-1501-9914)
- Mianxin Liu (ORCID: https://orcid.org/0000-0001-5171-778X)
- Shaoting Zhang (ORCID: https://orcid.org/0000-0002-8719-448X)
- Jie Xu (ORCID: https://orcid.org/0000-0001-5311-4704)
- Xiaoyang Li (ORCID: https://orcid.org/0000-0002-7418-4506)
- Xuanjing Huang (ORCID: https://orcid.org/0000-0001-9197-9426)
- Xiaofan Zhang (ORCID: https://orcid.org/0009-0005-2962-3644)
- Pengfei Liu (ORCID: https://orcid.org/0009-0008-6932-7091)
- Shanshan Wang (ORCID: https://orcid.org/0000-0002-0575-6523)
- Kang Li (ORCID: https://orcid.org/0009-0006-7318-2185)
- Jinru Ding
- Haitao Song (ORCID: https://orcid.org/0009-0001-5404-5154)
- Xin Sun (ORCID: https://orcid.org/0009-0003-7223-5298)
- Haofen Wang (ORCID: https://orcid.org/0009-0002-0145-7873)
- Zhian Bai
- Lifeng Zhu (ORCID: https://orcid.org/0000-0002-9231-2786)
Institutions
- Tongji University (CN)
- East China University of Science and Technology (CN)
- Shanghai Jiao Tong University (CN)
- Chinese Academy of Sciences (CN)
- Sichuan University (CN)
- Fudan University (CN)
- XinHua Hospital (CN)
- Ruijin Hospital (CN)
- West China Hospital of Sichuan University (CN)
- Beijing Academy of Artificial Intelligence (CN)
- Chinese University of Hong Kong, Shenzhen (CN)
- Shenzhen Institutes of Advanced Technology (CN)
- Shanghai Artificial Intelligence Laboratory (CN)
Publication Details
- Journal
- Big Data Mining and Analytics
- Published
- 2024-12-01
- DOI
- https://doi.org/10.26599/bdma.2024.9020044
- Citations
- 19
- Primary Topic
- Machine Learning in Healthcare
- Type
- article
- Field-Weighted Citation Impact
- 6.60
Funders
- National Key Research and Development Program of China