68 Use of artificial intelligence to derive International Metastatic Renal Cell Carcinoma Database Consortium (IMDC) elements from unstructured medical records

Abstract Background Artificial intelligence technologies invoking large language models (LLMs) may be able to automate data collection for the IMDC registry, an otherwise labor-intensive process. We evaluate a proprietary tool (HopeLLM) in expediting data abstraction. Methods We utilized a City of Hope-managed IMDC data set that includes patients with metastatic renal cell carcinoma. From this data set, we randomly selected patients who initiated care after 2018 and had sufficient treatment-related information within the Epic electronic health record system to determine HopeLLM performance. Patient identification numbers were manually extracted to prompt HopeLLM for data abstraction. We compared treatment start and end dates between manually registered information and HopeLLM estimations. The difference ratio in months was compared between the two sources. Lin’s concordance correlation coefficient was performed to quantify the rate of agreement between both sources. Results Among 513 records, 91 patients were randomly selected. A total of 164 lines of treatment were recorded manually in the registry, and 146 were recorded by HopeLLM. A total of 75 lines of treatment were unique to the registry, while 42 were unique to HopeLLM. There were 117 unmatched lines of treatment between the registry and HopeLLM, which were not considered for concordance analysis. We recorded a difference (median (IQR)) of 0 mos (0-0.09 mos) for start dates between the registry and HopeLLM, with a concordance correlation coefficient of 0.99 (95%CI 0.99-0.99). We recorded a difference of 2.15 mos (0.46-5.54 mos) for end dates between the registry and HopeLLM, with a concordance correlation coefficient of 0.92 (95%CI 0.88-0.95). Conclusions HopeLLM can streamline data abstraction from unstructured medical records onto the IMDC database, as it accurately predicts treatment start and end dates for patients with metastatic renal cell carcinoma.