Validating predictive models on EHRs data : lessons learnt from the NCDR-IMPACT-score applied on a French cardiac catheterization cohort

OBJECTIVE: We present a natural language processing pipeline to extract and re-compute a pre-existing score, the IMPACT-score, in a children's hospital belonging to the APHP hospitals. MATERIALS AND METHODS: Predictive variables and adverse events occurrences were extracted from the EHR of each patient. Data extraction involved rule-based and machine learning approaches depending on the format of the data. The machine learning text-classifiers were trained on active learning annotated dataset. Once the registry was automatically populated, we computed the IMPACT-score model in our patients and we performed a logistic regression analysis to find the specific odd ratio fitting our cohort and obtain the IMPACT-score-Necker. RESULTS: We extracted clinical data from 2,980 patients. When applied to our hospital cohort, the IMPACT-score-Necker achieved an AUC of 0.719 whereas the original IMPACT-score achieved an AUC of 0.642. As a reminder, the IMPACT-score achieved an AUC of 0.752 in the NCDR-IMPACT validation cohort. DISCUSSION: Local calibration of the IMPACT-score on our cohort enhanced predictive accuracy, improving the AUC from 0.642 to 0.719, and addressing differences between our population and the original NCDR-IMPACT cohort. This reinforces the need for model adaptation to local data, as patient demographic and clinical variations can significantly impact performance. Local EHR data warehouses could be leveraged for recalibration and continuous monitoring, ensuring that AI tools remain accurate, ethical, and practical for clinicians in their clinical practice. CONCLUSION: Our results underscore the need for real-world model validation, and EHRs offer a reliable source for training and validating AI models.