23 Generative artificial intelligence for automated unstructured MRI data extraction in prostate cancer care

Objectives/Goals: Magnetic resonance imaging (MRI) reports are stored as unstructured text in the electronic health record (EHR), rendering the data inaccessible. Large language models (LLM) are a new tool for analyzing and generating unstructured text. We aimed to evaluate how well an LLM extracts data from MRI reports compared to manually abstracted data. Methods/Study Population: The University of California, San Francisco has deployed a HIPAA-compliant internal LLM tool utilizing GPT-4 technology and approved for PHI use. We developed a detailed prompt instructing the LLM to extract data elements from prostate MRI reports and to output the results in a structured, computer-readable format. A data pipeline was built using the OpenAI Application Programming Interface (API) to automatically extract distinct data elements from the MRI report that are important in prostate cancer care. Each prompt was executed five times and data were compared with the modal responses to determine variability of responses. Accuracy was also assessed. Results/Anticipated Results: Across 424 prostate MRI reports, GPT-4 response accuracy was consistently above 95% for most parameters. Individual field accuracies were 98.3% (96.3–99.3%) for PSA density, 97.4% (95.4–98.7%) for extracapsular extension, 98.1% (96.3–99.2%) for TNM Stage, had an overall median of 98.1% (96.3–99.2%), a mean of 97.2% (95.2–98.3%), and a range of 99.8% (98.7–100.0%) to 87.7% (84.2–90.7%). Response variability over five repeated runs ranged from 0.14% to 3.61%, differed based on the data element extracted (p Discussion/Significance of Impact: GPT-4 was highly accurate in extracting data points from prostate cancer MRI reports with low upfront programming requirements. This represents an effective tool to expedite medical data extraction for clinical and research use cases.