Temporal drift, calibration monitoring, and adaptive updating of an IVIG resistance prediction model in Kawasaki disease: a multicenter pediatric cohort study

Clinical prediction models can lose accuracy over time as case mix and clinical environments change, but evidence on temporal drift and model updating in pediatric prediction models remains limited. We evaluated temporal drift in a multicenter prediction model for intravenous immunoglobulin resistance (IVIGR) in Kawasaki disease (KD) and compared recalibration, annual updating, and time aware updating strategies. We conducted a multicenter retrospective cohort study of children with KD treated between 2006 and 2024. Models were developed in the pre-COVID cohort (2006–2019) and validated in the during-COVID (2020–2022) and after-COVID (2023–2024) cohorts. Five algorithms were compared, and random forest (RF) was selected for subsequent recalibration and updating analyses. Performance was assessed using the area under the receiver operating characteristic curve (AUC), calibration, Brier score, and decision curve analysis. Strategies included temporally separated recalibration, all history refitting, half-life weighting, rolling windows, and calendar year adjustment. Analyses used 10 imputed datasets, and uncertainty was estimated using 500 bootstrap resamples. Among 6443 children (pre-COVID, 3788; during-COVID, 1837; after-COVID, 818), IVIGR occurred in 7.0%, 13.9%, and 10.1%, respectively. The RF AUC decreased from 0.776 during COVID to 0.629 after COVID, while its calibration slope decreased from 0.894 to 0.431. All history refitting achieved a mean AUC of 0.787 and a mean Brier score of 0.070. Calendar year adjusted RF and rolling window RF with a 3 year window both achieved a mean AUC of 0.804. Calendar year adjusted RF had the lowest Brier score, an observed to expected ratio closest to 1, and slightly higher net benefit across most thresholds. In this multicenter pediatric cohort, the IVIGR model showed temporal drift, with deterioration in discrimination and calibration and substantial variation across calendar years. Calendar year adjusted RF showed one of the most favorable overall performance profiles among the evaluated time-aware strategies. These findings indicate that the performance of IVIGR prediction models in KD may change after development, highlighting the importance of ongoing performance monitoring and model updating before and during long term clinical implementation.