Human knowledge-enhanced large language model agent for prediction of intestinal disease progression in patients with Crohn's disease: A multicenter retrospective study

Effective tools for risk stratification of intestinal disease progression (IDP) in Crohn's disease (CD) patients remain limited. Electronic medical records (EMRs) contain detailed information describing disease characteristics. Large language model (LLM) excels at summarizing and analyzing these data. We developed an LLM-agent using EMRs to predict IDP. In this retrospective study, we collected EMRs from 563 patients across six centers, built a human knowledge base incorporating literature-derived and EMRs-derived knowledge. Three levels of prompts were developed through a step-by-step superposition strategy: basic task description, Chain-of-Thought technique, and "Self-Reflection" module. The third level further integrated the human knowledge base. Three prompt versions (V1-3) were paired with three LLMs (ChatGPT-4omini, Gemini-1.5-Pro, ChatGPT-4o), creating nine LLM-agents. We compared their predictive performance for IDP to select the optimal model and contrasted the readability and semantics of LLM-generated reports with physician-written summaries. After evaluating 304 initial factors using a human-in-the-loop strategy with LLM and 13 CD specialists, we selected 98 IDP-related factors to construct the knowledge base. GPT-4o_V3 outperformed eight competing models in training set (accuracy, 84.4% vs. 53.9%-75.8%; F1 score: 0.867 vs. 0.663-0.808) and test set (accuracy, 83.5% vs. 59.8%-72.2%; F1 score, 0.882 vs. 0.748-0.814). Disease-progression-free survival of high-risk patients identified by GPT-4o_V3 was significantly shorter than low-risk patients ( P <0.001). Prediction reports generated from GPT-4o_V3 showed intermediate characteristics between Gemini-1.5-Pro_V3 and GPT-4omini_V3 in both readability and semantics. In conclusion, GPT-4o_V3 outperforms other models and accurately predicts IDP risk for CD patients across multiple medical centers, assisting clinicians in clinical decision-making. Electronic medical records (EMRs) offer comprehensive insights into disease characteristics. CrohnChatbot, our advanced large language model (LLM)-based agent, demonstrates exceptional capabilities in summarizing and analyzing such data. By integrating a curated human knowledge base, developed through the synthesis of high-quality literature and EMRs via LLM, CrohnChatbot's text analytical performance is significantly enhanced. This enables the agent to precisely identify high-risk Crohn’s disease patients who may benefit from early intensive treatment. Consequently, this not only supports clinicians in making evidence-based decisions and improving patient outcomes but also streamlines clinical workflows and enhances operational efficiency. • Electronic medical records (EMR) contain detailed information describing disease characteristics. Large language model agent (CrohnChatbot) excels at summarizing and analyzing EMR. • CrohnChatbot textual analytical ability could be effectively enhanced through our human knowledge base, assists clinicians in making informed clinical decisions and improves outcomes. • CrohnChatbot accurately identifies high-risk patient requiring early intensive treatment, improves clinician workload and overall efficiency.