Abstract PS3-06-23: Abstracting Lines of Therapy in Breast Cancer using Large Language Models

Abstract Background: Breast cancer research relies on accurate data from the electronic medical record (EMR). Manual chart review is time-consuming and error prone. To address this, we sought to use large language models (LLMs) to extract treatment data for complex patients with breast cancer. If successful, this approach will increase the data available for breast cancer research. Methods: We randomly selected 100 patients who had ≥ 2 lines of treatment and were diagnosed between 2010 - 2022 from an institutional database. To establish a ground truth, the medical oncologists on the team chart reviewed patients using all available data in the EMR. Research coordinators separately abstracted the data for comparison. We extracted the drugs used, the number of therapy lines, treatment intent (curative/palliative), start/stop dates, and the reason for discontinuation. We then developed an open-source pipeline using commercial HIPAA-compliant LLMs. We created text-delimited files containing either (1) the most recent oncology note, (2) Beacon chemotherapy plans, (3) medication lists, or (4) all these materials. We did not provide instructions on source prioritization in the prompt. We evaluated the performance of three LLMs: OpenAI's ChatGPT-4.1, an older ChatGPT, and Google Gemini 2.5, using cold temperature and default settings with sufficient token windows for a single API call. Outcomes examined were (1) identification of the exact anti-cancer drug(s), (2) the reason for stopping palliative intent therapy, and (3) palliative treatment duration ± 60 days. We report these outcomes as percentages correct, with the oncologist's chart review as the gold standard. Results: Among the 100 patients, 85% received curative therapy, and 89% received palliative therapy (median lines: 5; IQR, 3-7). The GPT-4.1 model, using a single oncology note, accurately identified all curative lines in 88% of patients (n = 85) and correctly identified 84% (n = 390 lines) of palliative intent lines. These results were similar for the Gemini 2.5 model (92% and 82%), and both outperformed an older version of ChatGPT (82% and 77%) and human research coordinators (78% and 73%). After excluding the final line of treatment, which may have been ongoing at the time of data censoring, GPT-4.1 and Gemini 2.5 identified the reason for stopping palliative intent therapy in 71% and 73% of lines where the model had already correctly identified the drugs. In comparison, the old ChatGPT was correct at 69%, while the research coordinators were correct at 56%. Adding unprocessed medication lists and Beacon plans to the note had a minimal impact on accuracy for all tasks (± 2%). Using Beacon or the med list alone led to poor accuracy in identifying lines of therapy, ranging from 0% to 39%. We reviewed the 5% of cases where all LLMs struggled. These were all patients who had some of their treatment with a different oncologist than the note author (e.g., second opinions). GPT-4.1 average process time per case was 12 seconds; Gemini 2.5 was 63 seconds. The oncologists required 15 - 75 minutes per case. Conclusions: LLMs can rapidly collect data from clinical notes with moderate accuracy, even without hyperparameter tuning, input standardization, or a foundational model. GPT-4.1 was more accurate than the research coordinators for all tasks. Our findings emphasize the importance of carefully considering inputs, prompts, model settings, and pipelines to ensure accurate variable abstraction. Further work is needed before researchers can use unsupervised abstraction at scale. Citation Format: J. C. Dickerson, M. Shaw, N. Dalal, M. McClure, N. Smith, J. Caswell-Jin. Abstracting Lines of Therapy in Breast Cancer using Large Language Models [abstract]. In: Proceedings of the San Antonio Breast Cancer Symposium 2025; 2025 Dec 9-12; San Antonio, TX. Philadelphia (PA): AACR; Clin Cancer Res 2026;32(4 Suppl):Abstract nr PS3-06-23.