Large Language Model Evaluation in Traditional Chinese Medicine for Stroke: Quantitative Benchmarking Study
This empirical study demonstrates that Chinese-centric models have a substantial advantage in static knowledge tasks within the TCM domain, whereas leading general-purpose models exhibit stronger dynamic reasoning and content generation capabilities. The TCM-SED, developed as the benchmark for this study, serves as an effective quantitative tool for evaluating and selecting appropriate LLMs for TCM scenarios. It also offers a valuable data foundation and a new research direction for future model optimization and alignment.
Authors
- Yang He (ORCID: https://orcid.org/0009-0000-2963-3897)
- Ji Bao (ORCID: https://orcid.org/0000-0001-8413-3270)
- Hongyan Long (ORCID: https://orcid.org/0000-0002-1773-046X)
- Yuzhu Zhang (ORCID: https://orcid.org/0009-0009-7087-2342)
- Yang Deng (ORCID: https://orcid.org/0000-0002-9677-3475)
- Yaoguang Guo (ORCID: https://orcid.org/0000-0001-8988-1577)
- Zhencai Shen (ORCID: https://orcid.org/0009-0003-9403-7284)
Institutions
- Sichuan University (CN)
- Southwest Medical University (CN)
- West China Hospital of Sichuan University (CN)
- Chengdu University of Traditional Chinese Medicine (CN)
Publication Details
- Journal
- JMIR Formative Research
- Published
- 2025-11-21
- DOI
- https://doi.org/10.2196/81545
- Primary Topic
- Traditional Chinese Medicine Studies
- Type
- article
- Field-Weighted Citation Impact
- 0.00