S1708 Ulcerative Colitis Advanced Therapy Timeline Prediction Using Large Language Models

Introduction: Accurately reconstructing medication timelines is essential across chronic diseases such as ulcerative colitis (UC), but manual chart review is laborious and structured data alone often misclassifies key dates. The start date is pivotal—it defines therapy sequencing and subsequent milestones (e.g., end of induction, one-year follow-up). We therefore developed a large language model (LLM)-based method that combines unstructured data and structured data to precisely predict each advanced therapy’s start date. Methods: We identified 193 UC patients seen at University of California, San Francisco (UCSF) Inflammatory Bowel Disease Center (2018–2023) and split them into training (n = 97) and test (n = 96) sets. Our goal was to predict the start dates of advanced therapies initiated at UCSF using structured inputs (orders, labs, administrations, procedures) and unstructured clinical notes processed by a HIPAA-compliant GPT-4o. Because our database only captures UCSF encounters, we first classified whether a medication was started at UCSF, then estimated its start date. Four models were compared against manual annotation: 1. Structured data via mixed-effects model; 2. Unstructured data processed by GPT-4o only; 3. Mixed-effects model combining structured data with GPT-4o–processed notes; 4. Combined structured and unstructured data processed by GPT-4o only. Results: Model 4 most accurately identified UCSF-initiated medications (sensitivity 94.4%, specificity 90.1%), outperforming models 1–3 (sens./spec.: 74.2%/71.6%, 79.8%/76.2%, 86.4%/81.2%). Among correctly classified UCSF starts, Model 4 also had the lowest mean absolute start-date error (30.3 days vs 119.1, 74.3, 73.5 days for models 1–3; P < 0.01). Models 2 and 3 both outperformed model 1 (P < 0.01) but did not differ from each other (P = 0.94). Using a one-month cutoff, Model 4 was accurate 90.9% of the time (vs 43.2%, 65.9%, 79.6%), and using a 2-week cutoff, Model 4 remained best at 81.8% (vs 38.6%, 68.2%, 61.4%). Conclusion: By processing structured data and unstructured clinical notes together only with an LLM, Model 4 minimized start-date prediction error more effectively than models using only structured data, only unstructured data, or a mixed-effects approach with both. It achieved over 90% accuracy within one month and over 80% within 2 weeks. While demonstrated in UC, this scalable method applies broadly to medication-use studies across diseases.