Comparison of large language model-processed unstructured clinical data versus manual physician review in breast cancer.

e13593 Background: Recent advancements in Large Language Models (LLMs) have been shown to automate clinical data extraction; however, their feasibility compared to traditional physician reviews remains understudied, particularly in complex areas such as breast cancer, where data heterogeneity poses significant challenges. Methods: In this retrospective comparative study, we analyzed breast cancer cases from five academic hospitals between January 2019 and December 2019. The cases were divided into two groups: manual physician review from electronic health records and LLM-based processing using Claude 3.5 Sonnet. We evaluated 89 clinical variables, including patient demographics, treatment information, and pathological characteristics. This study aimed to compare the efficacy of LLM-based processing with traditional physician reviews in organizing breast cancer clinical data, focusing on feasibility, consistency, and accuracy across multiple clinical parameters. Concordance was assessed using Cohen's kappa coefficient for categorical variables and Intraclass Correlation Coefficient (ICC) for continuous variables. The results were validated against the Korean Breast Cancer Society 2019 national registry data. Results: The manual physician review group (n=1,366) and LLM-based processing group (n=1,734) showed comparable performance in most data categories, including hormone receptor documentation (>90%) and surgical procedure distribution (manual vs. LLM in partial mastectomy: 63.5% vs. 63.9%). The LLM group demonstrated higher completion rates for lymph node assessment (78.5% vs. 91.2%) but more missing values in staging (3.1% vs. 12.2%). While ICC analysis showed low agreement for continuous variables, the stage distribution of the LLM group aligned better with national registry data. The LLM achieved 90.8% accuracy in validation analysis (817/900 factors), meeting the predetermined threshold of 90%, and captured more survival events (11 vs. 41) with significant outcome differences (HR=2.917, 95% CI: 1.496-5.688, p=0.002). Conclusions: LLM-based processing demonstrated comparable effectiveness to manual physician review in organizing breast cancer clinical data, with advantages in certain areas such as lymph node assessment and event detection. While showing potential as a viable alternative to traditional manual reviews, the higher rate of missing stage assessment, which needs integrated analysis in the LLM group, suggests areas for improvement. These findings support the potential role of LLMs in clinical data processing, while highlighting the need for continued refinement.