Automated identification of radiotherapy treatment sites from unstructured physician notes

PURPOSE: Ambiguous or incomplete documentation is a recurrent bottleneck in radiation oncology workflows, leading to inefficiencies in communication and potential treatment delays. Large language models (LLMs) pose a solution to addressing these ambiguities without added burden to clinical staff. We aim to assess the effectiveness of Meta's open-source Llama 3.3 model in using physician consultation notes to isolate and classify anatomical treatment sites and create helpful extractive summaries for each patient. METHODS: Semi-structured interviews with five radiation therapists revealed that CT simulation orders lack the necessary details to acquire the appropriate image. A retrospective cohort of 100 patient notes was used for iterative prompt engineering. The final model was evaluated on an independent test cohort of 52 patient notes. The LLM's accuracy in identifying the treatment site was benchmarked against two human observers (a medical physicist and a physician) as well as the final delivered treatment plan (ground truth). The helpfulness and accuracy of the AI-generated summaries were also rated by both observers on a 5-point Likert scale. RESULTS: Llama 3.3 achieved a weighted accuracy of 94.2% [95%CI: 89.4%-98.1%] when compared to sites isolated by either observer. When compared to the sites isolated from the retrospectively delivered plans, the model reached a weighted accuracy of 92.3% [95% CI: 87.5%-97.1%]. The model classified the anatomical sites with a weighted accuracy of 96.2% [95%CI: 87.0% -98.9%]. The AI-generated summaries were highly rated by both observers (Observer 1: 4.96 [95%CI: 4.87-5.00] and Observer 2: 4.58 [95% CI: 4.38-4.73]). CONCLUSION: This pilot study provides foundational evidence that LLMs can classify data with high accuracy, achieve benchmarks comparable to human experts when isolating anatomical treatment sites, and produce clinically helpful summaries. Our results suggest that LLMs can be effectively integrated to streamline complex radiotherapy workflows in the clinic.