TCMEval-PA: a question-answering benchmark dataset for the prescription audit of Traditional Chinese Medicine

Although large language models (LLMs) have witnessed rapid development in medical applications, their capacities to support rational medication use and guarantee prescription safety remain insufficiently investigated-especially in tasks such as prescription audit, which plays a critical role in safeguarding both. This paper presents TCMEval-PA, a benchmark dataset for assessing the capabilities of LLMs in prescription audit of Chinese herbal medicines. The dataset comprises 328 choice questions, including 297 single-choice and 31 multiple-choice. All questions were designed and compiled through rule extraction from official documents and reviewed by licensed TCM physicians. TCMEval-PA comprehensively encompasses the key dimensions of prescription safety, including normativity (e.g., dispensing, decoction requirements, and regulations for special medicines) and appropriateness (e.g., contraindicated combinations and excessive dosages). The present study employed TCMEval-PA to assess several prevalent Chinese and English LLMs. This dataset can be utilized for the evaluation of LLMs and other artificial intelligence (AI) systems in TCM prescription safety scenarios and promotes research in intelligent auditing and decision support.