Abstract PS3-04-23: Artificial intelligence (AI)-generated electronic medical record summarization in breast oncology

Abstract Introduction: Pre-charting for oncology patients (pts) requires manual review of documentation, often with varying formats. AI trained to summarize records for pre-charting may enhance efficiency in providing consistent information to clinicians, thereby enhancing the quality of care. We developed and clinically validated an AI agent to summarize records of pts with breast cancer. Methods: A generative AI agent utilizing GPT-4o and Retrieval-Augmented Generation (RAG) for processing radiology, pathology, and consultation reports was developed on a secure platform. Prompt engineering was employed to guide GPT-4o in creating oncology-specific summaries. For 50 consecutive breast cancer pts, history of present illness summaries (HPIs) generated by 4 breast surgery physician assistants (PAs) were compared to AI HPIs of identical records. Two breast surgeons evaluated the AI HPIs by rating the organization (clarity’) and contextual relevance (relevance’) of the information on a 5-point Likert scale (1=strongly disagree, 5=strongly agree). They clinically validated the AI HPIs by comparing to the PA HPIs using a structured evaluation framework focused on accuracy (factually correct/incorrect) and completeness of information (‘complete’ if AI HPIs aligned with the key information that oncologists would anticipate in a PA HPI) across predefined domains: patient age, imaging findings (mammogram, ultrasound, MRI, biopsy modality), pathology findings (histology, receptor status), medical history, and pending workup. Results: The overall clarity and relevance of the AI HPIs were high (average clarity 4.01, average relevance 3.87). AI accurately stated pt age in 94% of HPIs, with discrepancies attributed to birthday timing since initial workup. All pts had a mammogram, 86% had an US, and 36% MRI. Accuracy of imaging report summaries varied by modality, with MRI showing lowest accuracy (83.3%) and completeness rates (44.4%) (Table). Among the 85 biopsies, the biopsy modality was missing in 39.8%; AI HPI stated ‘core biopsy’ rather than ‘stereotactic guided’, ‘US guided’ or ‘MRI guided’ core biopsy. Pathology reporting demonstrated high accuracy among the 85 biopsies: histology 97.6%, hormone receptors and HER2 receptor 89.8%, and tumor grade 95%. Inaccurate AI responses resulted when grade or receptor stain percentage were reported in ranges. Two HPIs misrepresented past medical history. Among 12 patients requiring additional workup, 41.7% of AI HPIs appropriately identified pending studies. Conclusion: RAG-enabled GPT-4o demonstrated a high level of accuracy in breast cancer record summarization for pre-encounter preparation. Human review is necessary, but AI shows promise for clinical integration. Iterative prompt engineering, integration into clinical workflow, and evaluating the effectiveness of AI-assisted breast oncology medical record summarization is ongoing. Citation Format: K. Park, C. A. Minami, L. N. Butler, J. Gittzus, K. McLean, A. Dunn, E. A. Mittendorf, T. A. King. Artificial intelligence (AI)-generated electronic medical record summarization in breast oncology [abstract]. In: Proceedings of the San Antonio Breast Cancer Symposium 2025; 2025 Dec 9-12; San Antonio, TX. Philadelphia (PA): AACR; Clin Cancer Res 2026;32(4 Suppl):Abstract nr PS3-04-23.