S919 Large Language Models Combined With a Single Diagnostic Code Detect Eosinophilic Esophagitis in Electronic Health Records With High Accuracy

Introduction: Large-scale epidemiologic research for eosinophilic esophagitis (EoE) is hampered by variable accuracy of International Classification of Diseases (ICD) codes and case identification algorithms. We aimed to develop and validate a natural language processing (NLP) pipeline using large language models (LLMs) to identify EoE features and diagnosis from unstructured text, comparing performance with ICD codes and ICD + LLM combination. Methods: We identified gastrointestinal pathology reports with any mention of “eosinophil” paired with preceding GI clinic notes. Three hundred randomly selected patients were divided into training (N = 200, 56 with EoE) and testing (N = 100, 36 with EoE) sets. Manual chart review was used to assign an EoE reference standard. LLM prompt development used a human-in-the-loop approach with iterative refinements. Training concluded once the LLM-assigned diagnosis exceeded the F1 score (harmonic mean of precision and recall) of ICD codes. Using the test set, we compared performance of ICD codes, LLM-derived diagnostic features, LLM-assigned diagnosis, and a combined LLM + ICD. Performance metrics included sensitivity (recall), PPV (precision), and F1 score. Nonparametric bootstrap resampling (1,000 replicates) was used to estimate 95% confidence intervals (CIs) with statistical significance based on whether they excluded zero. Results: In the training set, LLM-derived diagnostic features demonstrated the highest sensitivity (0.98; 95% CI 0.95, 1.00), while LLM-assigned diagnosis had the highest PPV (0.92; 95% CI 0.84-1.0) and specificity (0.97; 95% CI 0.95,1.00). Comparably high F1 scores were achieved with LLM-derived features (0.89; 95% CI 0.83, 0.95) and ICD + LLM diagnosis (0.88; 95% CI 0.81, 0.95). In the independent test set, ICD codes alone showed a sensitivity of 0.86 (95% CI 0.75, 0.97), PPV of 0.97 (95% CI 0.91, 1.0), and an F1 of 0.91 (95% CI 0.84, 1.0). Combining ICD and LLM-assigned diagnosis yielded a 3-point improvement in F1 score (95% CI -0.01, 0.07; P = 0.2) compared to ICD alone. This combination method significantly improved sensitivity (0.92 [0.83,1.00]; P = 0.008) and F1 score (0.94 [0.89,1.00]; P = 0.047) relative to LLM-assigned diagnosis. Conclusion: Combining the LLM-assigned diagnosis with a single diagnostic code reduced false negatives and modestly improved the F1 score compared to either method alone, suggesting a scalable approach for improving EoE case identification in real-world data.