Large language models for automated scoring of subjective examination questions in a traditional Chinese medicine curriculum: a four-model comparative evaluation across three disciplines

Abstract Background. Manual scoring of subjective questions in traditional Chinese medicine (TCM) education is labor-intensive, time-consuming, and subject to inter-rater variability. Large language models (LLMs) offer potential for automated scoring, yet systematic multi-model, multi-discipline comparative evaluations remain scarce. Methods. Guided by agile data management methodology, we designed an intelligent scoring database and a five-dimensional evaluation framework (accuracy, reliability, stability, timeliness, and economy). Four LLMs—ChatGPT-4.0, DeepSeek-V3, ERNIE 4.0, and Tongyi Qianwen 2.5—scored 500 student responses to five questions across pharmacology, febrile diseases (TCM), and physiology using a structured scoring prompt framework. Model outputs were compared against expert raters using mean absolute error ratio, root mean square error, Spearman’s rank correlation coefficient, intraclass correlation coefficient (ICC), and Wilcoxon signed-rank tests; each model performed two scoring passes plus a third adjudication pass for discrepant cases. Results. Significant cross-model and cross-discipline differences were observed. Tongyi Qianwen 2.5 showed the most consistent performance across disciplines (ICC 0.615–0.917). ERNIE 4.0 excelled in pharmacology (MAER 0.087–0.170) but performed poorly in febrile diseases (MAER 0.257). ChatGPT-4.0 and DeepSeek-V3 performed strongly in febrile diseases and physiology. All four models struggled with a complex pharmacology mechanism question (cardiac glycoside positive inotropic action). Hallucination rates were low across all models (0.03%–0.74%). Scoring efficiency varied substantially: DeepSeek-V3 achieved the highest timeliness (mean < 7 s per response) and the lowest cost (99.8% economy score), whereas ChatGPT-4.0 incurred the highest cost (41.1% economy score). Conclusions. LLMs show promise for automated scoring of subjective examination questions in TCM education, but model selection should be discipline-specific. Optimizing prompt frameworks and integrating human-in-the-loop validation can further enhance scoring reliability. Agile data management methodology provides a scalable approach to educational data lifecycle management.