S27 Detection of Ultra-High Risk Pancreatic Cancer Diagnoses Using a Custom NLP Pipeline Incorporating a Large Language Model: A Pilot Safety Net for Missed Diagnoses

Introduction: Radiologic findings suggestive of malignancy may be missed or inadequately acted upon in large health systems, leading to delayed diagnoses. Natural language processing (NLP) offers the potential to surface high-risk cases buried in unstructured data. We developed a custom NLP pipeline that incorporates a commercial large language model (LLM) along with additional filtering components to flag radiology reports indicative of new, unexpected, or high-risk cancer findings. This pilot uses pancreatic cancer to evaluate whether such a system can uncover clinically significant yet overlooked diagnoses. Methods: We used NLP to create a custom pipeline using 589,513 radiology reports from a large health system. The model flagged 2,897 reports related to pancreatic cancer; 43 were identified as suspicious for new, unexpected, or particularly worrisome findings. We then conducted detailed manual review of 10 consecutively flagged cases to assess follow-up, diagnostic closure, and timeliness of response. Results: Among the 10 patients, 9 had findings concerning for new or progressive pancreatic cancer, and 1 had stable known disease. Gaps in follow-up were common: 2 patients lacked biopsy or additional imaging months after initial scan. Three had care refusal or communication breakdowns without documented contingency planning. Two presented emergently with symptoms but lacked outpatient coordination. One had cystic disease with malignant transformation and no interim evaluation. These ultra-high-risk scenarios occurred in fewer than 1 in 10,000 reports—illustrating the rarity and value of LLM-enabled detection. Conclusion: This pilot shows that an LLM-based pipeline, combined with targeted filtering, can detect rare but urgent radiologic findings that may otherwise be missed. Artificial intelligence-driven safety nets could help health systems identify overlooked malignancies and improve early cancer detection. These cases highlight both the potential for harm and the opportunity for intervention when actionable signals are surfaced early.