Leveraging large language models on personal computers for automated identification of lung cancer progression in radiology reports.

624 Background: Radiology reports are used to identify lung cancer treatment response for clinical care and research. Disease progression is typically documented as free text. Extracting instances of progression from numerous reports during chart review for clinical research requires time-intensive manual labor. Conversely, large language model (LLM) tools can quickly extract this information, but they demand significant computational resources such as graphics processing units (GPUs). We aimed to develop an LLM tool using structured “few-shot” prompting, in which examples are provided to optimize LLM output, to accurately identify lung cancer progression from radiology reports on a personal computer without a GPU. Methods: To generate a gold standard for review, we manually annotated 400 chest CT reports using RECIST 1.1 criteria. Each report listed measurements of all thoracic (i.e., lung, mediastinal, and chest wall) masses and indicated whether these lesions were smaller or larger than in the prior study. We tested three LLMs (Llama3-8B, Mistral-7B, and Phi3-4B) for mass size extraction, report summarization, and binary classification (stable vs. progression) using chain-of-thought reasoning. The model first extracted size changes for each lesion. The model was then queried to classify the report as indicating progression or stability, along with confidence scores. Predictions with low confidence scores were defaulted to stable. Models were compared using the same prompts, differing only in model architecture and parameter count. Results: 39% of reports indicated progression per gold standard labels. Among the models tested, Llama-3 achieved the best performance with an F1 score of 0.866 and overall accuracy of 82.5%. Of all the cases Llama-3 predicted as stable, 95.8% were actually stable. Mistral followed with F1 score of 0.807, an overall accuracy of 76.4%, and precision of stable predictions of 97.5%. Smaller models such as Phi-3 did not consistently provide the requested labels. Upon manual review of the cases incorrectly labeled by Llama-3, 60% of incorrect labels were applied to borderline cases (e.g., slight disease growth that did not meet RECIST criteria, or growth that the radiologist considered to be attributable to factors such as inflammation). Conclusions: LLMs utilizing few-shot prompting can reliably and efficiently identify lung cancer progression from free-text radiology reports on a personal computer. Llama-3, operating without GPU support, achieved high sensitivity and F1 scores, enabling accurate extraction of clinical outcomes from unstructured text. Locally deployed LLMs can enable scalable, cost-effective data abstraction for retrospective research and may offer practical tools for researchers without high-end computing infrastructure.