Abstract 4147468: Using Structured Queries versus Retrieval Augmented Generation Enabled GPT-4 to Phenotype Atherosclerotic Cardiovascular Disease

Background: Using Electronic Health Record (EHR) data to accurately phenotype patients for atherosclerotic cardiovascular disease (ASCVD) is crucial for clinical decision-making and research. Traditional methods using structured queries often lack specificity or sensitivity. We developed RECTIFIER, an AI-based clinical phenotyping tool. This study compares the performance metrics of structured queries against RECTIFIER in determining types of ASCVD. Methods: We conducted an analysis using EHR to compare the performance of structured queries with RECTIFIER for identifying various ASCVD types. The dataset included 998 patients with diabetes. Structured queries identified ASCVD types using ICD, CPT, and HCPCS codes, while RECTIFIER analyzed unstructured clinical notes within the previous year to identify the same components. Three expert clinicians adjudicated each ASCVD component by reviewing patients’ charts. Performance metrics, including sensitivity, specificity, and accuracy, were calculated for both methods. Results: Structured queries demonstrated similar or lower sensitivity in identifying ASCVD types compared to RECTIFIER (Figure). The sensitivity of structured queries to determine any ASCVD was significantly lower than that of RECTIFIER (Structured: 67.8%, RECTIFIER: 87.3%). Conversely, structured queries exhibited similar specificity with RECTIFIER except for greater performance in PCI and coronary revascularization which showed small but statistically significant differences (Structured: 100% and 95.1%, RECTIFIER: 98.1% and 90.1%, respectively). Conclusion: The higher sensitivity of RECTIFIER in identifying ASCVD components suggests its potential for improving patient phenotyping. However, the greater specificity of structured queries underscores its utility in specific procedural identification. An integrated approach, using RECTIFIER for general phenotyping and structured queries for procedure-specific components, may enhance the accuracy and efficiency of ASCVD identification. Future studies should explore optimized prompting strategies to further refine structured queries' performance and validate these findings in larger, diverse populations.