Enhancing pancreatic cancer staging with large language models: the role of retrieval-augmented generation

Retrieval-augmented generation (RAG) is an emerging technique that enhances large language models (LLMs) by retrieving relevant information from reliable external knowledge (REK). Despite its success in various language processing tasks, the clinical application of RAG in radiology is still novel. To evaluate the utility of RAG in radiological staging tasks, we compared the performance of NotebookLM, a RAG-equipped LLM (RAG-LLM), with its internal model, Gemini 2.0 Flash. Using Japan’s current pancreatic cancer staging guideline as REK, we compared three LLM settings—(1) NotebookLM with REK (REK+/RAG+), (2) Gemini 2.0 Flash with REK (REK+/RAG−), and (3) Gemini 2.0 Flash without REK (REK−/RAG−)—in staging 100 fictional pancreatic cancer cases based on CT findings. Staging tasks included assessment of TNM classification, local invasion factors, and resectability classification. The REK+/RAG+ group achieved 70% staging accuracy, outperforming REK+/RAG− (38%) and REK−/RAG− (35%). For TNM classification, REK+/RAG+ reached 80% accuracy, compared to 55% and 50% in REK+/RAG − and REK−/RAG−, respectively. NotebookLM also presented retrieved REK excerpts as rationale, with a retrieval accuracy of 92%. These results suggest that the RAG system implemented in NotebookLM improves LLM staging performance by enabling access to up-to-date medical guidelines. Furthermore, its ability to retrieve and present source evidence enhances transparency and allows users to verify the reliability of model outputs. This study highlights the potential of a specific RAG system (NotebookLM) to support pancreatic cancer staging by combining clinical language interpretation with direct reference to authoritative guidelines, in an idealized and controlled proof-of-concept experimental setting.