Abstract 7427: Use of large language models for clinical data abstraction from oncologic medical records

Abstract Introduction: Abstraction of clinical data from unstructured medical records is a labor-intensive process. Large language model (LLM)-based systems can accelerate the availability of data for research use. Participants and Methods: The American Cancer Society’s Cancer Prevention Study-3 is a large, nationwide prospective cohort study of ∼300, 000 cancer-free participants enrolled between 2006-2013. Participants who self-reported a diagnosis of breast, colorectal, ovarian, or any blood cancer on the 2021 follow-up survey were consented for the retrieval of medical records (available as a single PDF) from diagnostic and treatment facilities. Guidelines, e.g. ontologies and decision trees, were generated to precisely define abstraction tasks, and relevant research data were annotated via human abstraction. This project then used 300 breast cancer-related medical records to develop (n=200) and test (n=100) the performance of Layer Health’s LLM-based platform Distill to abstract seven data elements from the medical record: cancer behavior, laterality, neoadjuvant therapy status, and presence of key biopsy or surgery with associated procedure dates. Both an answer and corresponding evidence from the record were provided by their algorithms; this evidence was used to conduct quality control on a small fraction of data points that were automatically flagged by inter-variable consistency checks. An ACS oncology data specialist adjudicated disagreements between human and LLM-based abstraction to update ground truth labels. Accuracy and F1 statistics were then calculated between ground truth and LLM-derived results. Results: Results on the test set (n=100) were available on the same day; adjudication of 2​5​​ discrepancies between human and LLM-based abstraction found 12 LLM errors and 13 human errors. Cancer behavior and laterality were abstracted with 100% accuracy​ by the LLM system​. Neoadjuvant therapy (yes/no) was abstracted with 99% accuracy (F1=0.99). Surgical and biopsy procedures (yes/no) were abstracted with 98% (F1=0.99) and 96% (F1 = 0.98) accuracy, respectively. 94% of identified key biopsy dates were accurate within 1 day, and 96% were accurate within 30 days. 96% of identified key surgery dates were accurate within 1 day, and 97% were accurate within 30 days. Conclusion: LLM-based systems offer a time-efficient and accurate solution to accelerate abstraction of key data elements for use in oncologic research, including ​complex multi-step ​variables. Citation Format: Jillian Nelson, Monica Agrawal, Den E Bloodworth, Divya Gopinath, James Mullenbach, Peter J. Briggs, Dominique Connolly, David Sontag, Alpa V. Patel. Use of large language models for clinical data abstraction from oncologic medical records [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 7427.