TNM Classification of Pancreatic Cancer From Unstructured Radiology Reports Using a Reasoning Language Model

Purpose: The ability of large language models (LLMs) to determine the tumor, node, and metastasis (TNM) classification of cancer from radiological reports based on simple prediction is limited. This study aimed to evaluate the performance of a reasoning language model (RLM)-an LLM with critical thinking, logical deduction, and multi-step reasoning-in identifying the TNM classification of cancer from unstructured radiological reports. Materials and Methods: We retrospectively screened 100 consecutive radiology reports, written in Japanese, from computed tomography (CT) scans for pancreatic cancer conducted between April 2020 and June 2022. The cohort included 62 male and 38 female patients, with a mean age of 70.3 ± 10.8 years. OpenAI o1, the first RLM, was used to classify the TNM staging from the radiology reports based on the General Rules for the Study of Pancreatic Cancer, seventh edition. The accuracy and kappa coefficients of the TNM classifications by o1 were assessed and compared with the evaluations of two board-certified radiologists. Results: The accuracy values for T, N, and M classification were 0.87, 0.99, and 0.97, respectively. The corresponding kappa coefficients were 0.74, 0.97, and 0.93. Conclusions: The reasoning language model accurately determined the TNM classification for pancreatic cancer from unstructured Japanese radiology reports.