Identifying distant metastatic breast cancer recurrence: Validity of large language model and discrete-field algorithm approaches versus manual chart review.

578 Background: Reliably identifying whether a patient has distant cancer recurrence and determining the timing of the recurrence has been a challenge limiting metastatic cancer research. Manual chart review is the gold standard for identifying this information; however manual review is burdensome and time intensive. Ritzwoller et al. previously developed and validated a discrete-field algorithm (Ritzwoller algorithm) for identifying recurrence and recurrence timing. With advances in HIPAA-compliant large language models (LLMs), we aimed to characterize the reliability of an LLM-based approach versus gold standard chart review and the Ritzwoller algorithm in identifying metastatic breast cancer recurrence. Methods: We used clinical notes and electronic health record data from January 2010 to December 2023 to evaluate the performance of two approaches for identifying distant metastatic breast cancer recurrence.: an algorithm that assesses imaging, drugs and encounters (the Ritzwoller algorithm) and an LLM using GPT-4o (version 2024-02-15) to process clinical notes. Both were assessed against manual chart review conducted by at least two clinical experts (gold standard). To optimize cost and computation, the LLM approach first applied a keyword filter to retain only clinical notes containing “metasta”. GPT-4o, accessed via UTSW’s private Azure OpenAI service, processed notes by year in ascending temporal order: once classified as distant metastatic, the patient’s subsequent notes were excluded. Results: Our cohort included 512 patients initially diagnosed with stage I-III breast cancer. In total, 10,924 notes out of ~315,000 were reviewed by GPT-4o, with the majority (71.8%) being progress notes, mostly authored by physicians (64.6%). The full LLM process was completed in 10 hours at an estimated cost of $80. Compared to gold-standard manual chart review, GPT-4o achieved a sensitivity of 0.95, specificity of 0.80, positive predictive value (PPV) of 0.68 and negative predictive value (NPV) of 0.91. The Ritzwoller algorithm had negligible processing time, and a sensitivity of 0.79, specificity of 0.98, PPV of 0.95, and NPV of 0.91 compared to chart review. GPT-4o demonstrated higher sensitivity and NPV but lower specificity and PPV. Conclusions: Reducing the number of notes with a keyword pre-processing significantly reduced LLM cost and achieved strong performance. Current LLM performance is better suited for research questions benefiting from more inclusive case capture but could be improved with prompt engineering and refined note selection.