A living systematic literature review (L-SLR) for non–small-cell lung (NSCLC), prostate (PC), and breast cancer (BC), built with an agentic text annotation system powered by large language models (LLM) to assist treatment decision making.

e13657 Background: With demanding schedules, oncologists often struggle to stay updated on the latest clinical trial data from publications, especially without access to pre-curated databases. Traditionally, building annotated clinical trial databases required extensive time and manual effort. To address these challenges, we explored assembling an L-SLR library of clinical trial data using extractions generated by an agentic LLM system and evaluated the system’s accuracy and associated time savings. Methods: Agentic LLM systems are autonomous systems where multiple LLMs maintain control over how they accomplish tasks with no human input or supervised training. Our system used three OpenAI LLM models (o1, o3, o1-mini) in a matrix of processes to emulate trained human experts by following an annotation manual, subdividing complex processes into smaller subtasks, and documenting its reasoning for traceable results. This methodology is inspired by the recent development from DeepSeek reinforcement learning and reasoning capabilities. Annotations were created for 4 review variables (population, intervention/comparator, outcome, study design) and 32 extraction variables, including clinical TNM staging, histology, biomarker, associated risk factors, treatment path (line, prior therapy etc.), interventions, intervention type; study randomization, phase and sample size; analysis type, follow-up period; reported outcomes (median and landmark overall survival (OS), progression-free survival (PFS) and other progression measures, response data, quality of life data); subgroup analyses, safety/toxicity. Accuracy of review and extraction was evaluated on publications in three cancer types: NSCLC, PC, BC compared to human results. Results: Our agentic LLM system generated annotations for 4 review variables for 19,407 publications (6,916 NSCLC, 6,978 PC, 5,513 BC) publications, and 32 extraction variables for 2,424 (1,356 NSCLC, 587 PC, 481 BC) publications. Accuracy for the review variables ranged from 93.8% to 97.2%. For extraction variables, accuracy exceeded 90% for all variables (91.5%-99.1%) with 50% of variables above 95%. Our system completed the annotations in 5.34 hours, compared to an estimated 727.39 hours by trained human researchers, resulting in 99.27% time savings. Conclusions: Our living SLR system can accurately review and extract clinical trial publications with performance comparable to human experts. This level of accuracy highlights our system’s potential to deliver real-time clinical data, empowering oncologists to make more informed treatment decisions, with the hopes of ultimately improving patient outcomes.