Targeted use of large language models for EHR-based computable phenotyping

OBJECTIVE: Computable phenotypes derived from electronic health records (EHRs) are central to clinical research and quality reporting. Although large language models (LLMs) can extract clinically rich information from unstructured notes, routine application to all patients is computationally expensive. We evaluated whether uncertainty-guided selective use of LLMs can improve phenotyping accuracy while preserving scalability. MATERIALS AND METHODS: We developed a selective augmentation framework integrating structured and unstructured EHR data using uncertainty-guided triage. An ensemble of heterogeneous classifiers trained on structured data generated probabilistic phenotype predictions and uncertainty measures to identify patients at elevated risk of misclassification. Only flagged patients underwent LLM-based analysis of unstructured clinical notes using retrieval-augmented generation. LLM-derived outputs were incorporated as additional predictors in a final probabilistic model. Performance was evaluated for two registry-based phenotypes: diabetes mellitus and peripheral arterial disease (PAD), using internal cross-registry and external validation cohorts. RESULTS: For diabetes mellitus, selective augmentation improved sensitivity in the internal validation cohort from 0.81 to 0.90 without loss of specificity (0.92). More than 70% of triage-flagged patients represented misclassifications by structured data alone. For PAD, selective augmentation markedly increased sensitivity from 0.18 to 0.97 while maintaining high specificity (0.99), requiring LLM analysis for only 10% of patients. DISCUSSION: Uncertainty-guided triage efficiently concentrated LLM use on patients most likely to benefit, improving case identification-particularly for phenotypes poorly captured by structured data-while minimizing computational burden. CONCLUSION: Selective, uncertainty-guided integration of LLMs enables scalable, interpretable, and accurate EHR-based phenotyping, offering a practical alternative to universal LLM deployment in real-world informatics workflows.