In vivo clinical effectiveness of artificial intelligence screening for acute coronary syndrome: testing real-world performance

Objective Many predictive models are being developed, but few are deployed in clinical care. Hence, model performance within live real-world care is often not known. We trained a predictive model (in vitro) to (1) estimate patients’ risk of acute coronary syndrome (ACS) on arrival to the emergency department (ED) and (2) identify those with highest risk for ST-segment myocardial infarction (STEMI) to receive an early ECG. We embedded this model into clinical care as clinical decision support (CDS) and using dynamic real-world data (in vivo), ran a silent pilot. We aimed to test the CDS’s replication of the original model’s screening performance and compare to standard of care screening by human staff. Methods and analysis The CDS prospectively assessed each patient arriving in the ED between November 2023 and April 2024. It calculated each patient’s risk for ACS, recorded a decision about whether the patient should receive an early ECG and was programmed to not exceed the total number of ECGs performed in human practice, approximately 33%. We used raw agreement and Cohen’s Kappa to compare the screening decisions of the in vivo CDS and original in vitro model. We then measured sensitivity for ACS, our primary outcome, and specificity, which we compared between the CDS’s and human screening decisions. Results 32 346 visits were seen in the ED and processed by the CDS. 1.0% had ACS and 0.1% STEMI. Raw agreement between the CDS and original model was 96.8%, and Kappa was 91.2% (95% CI 90.7% to 91.8%). Sensitivity for ACS was 81.7% (95% CI 77.1% to 85.8%) in CDS versus 80.2% (95% CI 75.4% to 84.4%) observed in humans. Specificity for ACS was 67.3% (95% CI 66.8% to 67.9%) versus 67.4% (95% CI 66.9 to 67.9%), respectively. Conclusion We found very good raw agreement and Kappa scores between the CDS and original model. Sensitivity and specificity between the CDS and human practice did not differ. This suggests that, when CDS is deployed independent of human decision-making, we can expect very high, although not perfect, performance replication. These differences need to be quantified and considered to estimate the real-world impact of models before deployment.