A Large Language Model–Driven System for Advance Care Planning Training Among Health Care Providers in the Chinese Context: Development and Technical Evaluation

Abstract Background With the expanding need for advance care planning (ACP), innovative educational strategies for training health care providers are increasingly required. Large language model (LLM)–based ACP chatbots offer a novel and potentially effective solution to enhance health care providers’ competence in navigating complex ACP conversations. Objective This study aimed to develop a Chinese-context ACP corpus to support an LLM-based chatbot and evaluate the feasibility and performance of a multi-agent system for simulating complex ACP discussions as a training tool for health care providers. Methods This study involved dataset construction and model adaptation and evaluation. We constructed 3 structured datasets using synthetic dialogue data generated through prompts derived from ACP-related scientific literature and policy documents. Both open-source (Zhongjing) and closed-source LLMs (GPT-4o-mini) were chosen as baseline models. The Zhongjing model was adapted through fine-tuning, whereas GPT-4o-mini was adapted using both fine-tuning and prompt engineering. Model performance was assessed through automatic and human evaluations following the QUEST (Quality of information, Understanding and reasoning, Expression style and persona, Safety and harm, and Trust and confidence) framework. Statistical comparisons between baseline and adapted models were performed using repeated-measures ANOVA. Results Three separate datasets for the assistant, vignette, and evaluator agents were created, which collectively formed a multi-agent artificial intelligence system for Chinese ACP training. The assistant dataset included 4364 dialogues, the vignette dataset comprised 671 clinical scenarios, and the evaluator dataset contained 671 records. Both automatic and human evaluations confirmed that the adapted models significantly outperformed baseline models on most aspects of Chinese ACP conversations and summarization (η 2 p =0.12‐0.99; P values ranged from .03 to <.001). Conclusions This study demonstrates the adequate technical feasibility of the multi-agent LLM-based system for ACP training among health care providers in the Chinese context. Despite its potential as a supportive educational tool, further validation in real-world training contexts is required to establish its effectiveness in enhancing health care providers’ ACP competencies.