Assessing the true burden of immune-related colitis in lung cancer through large language model–enabled clinical phenotyping.

1615 Background: Immune checkpoint inhibitors (ICIs) have revolutionized lung cancer treatment but cause adverse events, including immune-related colitis (ir-colitis). Accurate identification of ir-colitis is critical for improving patient outcomes and clinical research. However, current methods relying on ICD codes have substantial limitations: non-specific categories that fail to capture the cases or distinguish cases from infectious or ischemic etiologies, as well as incomplete coding that leads to missed cases. Large language models (LLMs), with advanced natural language understanding capabilities, offer a scalable approach for automated phenotyping using unstructured clinical notes. We aimed to develop and evaluate an LLM-based approach for identifying ir-colitis in patients with lung cancer receiving ICIs and compare it with ICD code based method. Methods: We identified 5,278 lung cancer patients with ICI administration at Mayo Clinic (2012-2025). To investigate ir-colitis occurrence within two years after ICI initiation, patients were stratified into three ICD-based cohorts: (1) IR-Colitis, with clinician-assigned ir-colitis-specific diagnosis codes; (2) Non-IR Colitis, with colitis diagnosis codes excluding ir-colitis; and (3) No-Colitis, absent of any colitis-related codes. We used three open-source LLMs to process clinical notes : Llama4, MedGemma, and Qwen. Prompts were designed to extract both gold standard evidence (explicit ir-colitis terminology, such as "immune-mediated colitis" and "checkpoint inhibitor colitis") and supportive evidence (such as pathology and lab results). Results: Llama4 demonstrated superior accuracy among three LLMs in manual evaluation and was thus selected for all cohort analyses. Among ICD-based IR-Colitis patients (n = 339), 82.0% had confirmatory evidence while 18% lacked evidence, suggesting potential false-positive coding. In the Non-IR Colitis cohort (n = 297), LLM-based extraction identified 43.8% patients in fact had ir-colitis, demonstrating substantial misclassification. In the No-colitis cohort (n = 4,642, random sample of 996 for analysis), the LLM identified 10.7% with documented ir-colitis, revealing substantial missing by administrative coding. Conclusions: LLM-based phenotyping demonstrates superior sensitivity for identifying ir-colitis compared to ICD codes alone, successfully uncovering cases missed by administrative coding. This scalable approach enables more comprehensive case identification for immunotherapy safety research and clinical surveillance. ICD-Based Cohorts LLM-Identified ir-Colitis Gold Standard Evidence Supportive Evidence Any Evidence ir-Colitis (n = 339) 277/339 (81.7%) 275/339 (81.1%) 278/339 (82.0%) Non-ir-Colitis (n = 297) 129/297 (43.4%) 126/297 (42.4%) 130/297 (43.8%) No Colitis (n = 996) 106/996 (10.6%) 104/996 (10.4%) 107/996 (10.7%)