Abstract LB450: AI-assisted pathology report abstraction for breast cancer

Abstract Background: Clinical cancer research often involves time-consuming manual abstraction of tumor information from pathology reports. Large language model (LLM)-based abstraction offers an efficient alternative but lacks transparency due to the “black-box” nature of LLMs. We evaluated the speed and accuracy of a combined approach using BRIM, an LLM-assisted human-in-the-loop clinical abstraction tool. Methods: We transcribed text from 828 PDF breast tumor pathology reports (399 previously abstracted by a trained tumor registrar) for 588 participants in the Carolina Breast Cancer Study, Phase 4 (CBCS4). We designed prompts to abstract tumor size, grade, estrogen and progesterone receptor (ER, PR) and human epidermal growth factor receptor 2 (HER2) information from reports, including percent positivity, staining intensity, immunohistochemistry (IHC) scores, as appropriate. Prompts were tuned through iterative testing in BRIM in conjunction with GPT-OSS-20B (an open-source LLM), where abstractors reviewed in-text evidence and model reasoning, providing real-time feedback to improve subsequent outputs. Performance metrics included abstraction speed and accuracy compared to gold-standard manual abstraction. Only cases with available data from both the LLM and registrar were included. Results: LLM-assisted abstraction of the 828 notes took 11.5 hours compared to 100 hours by our certified tumor registrar (average 7 min/report), representing a 90% reduction in abstraction time. As shown in Table, abstraction accuracies ranged from 85% (% positivity for PR) to 98% (tumor grade, ER intensity, and ER categorical interpretation), with most accuracies exceeding 90%. Conclusion: LLM-assisted abstraction of tumor variables from pathology reports is feasible, accurate, and may substantially reduce the burden of manual abstraction. Further work incorporating additional complexity (e.g. addenda, multifocality) will be important for more closely mimicking registrar workflows. Citation Format: Sarah C. Van Alsten, Saianand Balu, Isaiah W. Zipple, Georgia C. Mudd, Nader Mehri, Daniel Fabbri, Melissa A. Troester. AI-assisted pathology report abstraction for breast cancer [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2026; Part 2 (Late-Breaking, Clinical Trial, and Invited Abstracts); 2026 Apr 17-22; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2026;86(8_Suppl):Abstract nr LB450.