Auditing imbalanced hospital readmission prediction: patient separation, resampling contamination and probability calibration

Abstract Hospital readmission models require evaluation that respects repeated patients and separates training from validation. We audit a workflow predicting recorded readmission within 30 days and reanalyze 59,129 encounters from 39,726 patients using patient-disjoint fitting, calibration, and evaluation partitions. In the source split, 46.08% of test encounters belonged to patients also represented in training. A controlled 20-partition experiment additionally compares fold-local synthetic oversampling with a contaminated donor pool while holding real validation observations and synthetic training counts fixed. Under patient-level splitting, including validation donors changes mean cross-validation area under the receiver operating characteristic curve (AUC) by +0.005 for logistic regression and +0.012 for XGBoost. In the primary mixed-type analysis, logistic regression and XGBoost achieve evaluation AUCs of 0.653 and 0.657 and average precisions of 0.215 and 0.231. Their paired AUC difference is 0.0041 (95% patient-cluster interval: \((-0.0042)\) to 0.0128). Recall is 0.255 and 0.263 at thresholds chosen on calibration patients under a 10% false-positive-rate constraint. A prior-admissions-only reference reaches AUC 0.604. Lower categorical-aware synthetic oversampling ratios improve on full balancing but do not exceed the corresponding unresampled baselines; retaining laboratory-result categories does not improve discrimination on the same evaluation patients. Exploratory subgroup estimates identify differences in recall, with limited support in small groups. Together, the controlled audits, calibration assessment, and sensitivity analyses provide a reproducible account of how evaluation choices shape performance in this historical readmission cohort.