What Data Audits Miss: Prediction Error and Comparison Coverage in a Controlled MIMIC-IV Study

Clinical datasets can have the same frequencies of missing laboratory measurements yet differ in which patients lack them. We tested whether data audits can accept such datasets as similar despite differences in prediction error, and how many comparisons each audit retains. Using MIMIC-IV, we created matched dataset versions for 10,800 development patients by removing measurements from four laboratory tests at random or preferentially from patients with higher or lower predicted risk. Risk priorities used earlier clinical information. The versions preserved the complete distribution of missingness patterns. At 40% additional omission, mean Brier loss (mean squared probability error) for a fixed boosting model was 0.0911 under higher-risk omission and 0.0865 under lower-risk omission (difference 0.0046; 95% context-bootstrap interval, 0.0035–0.0058). The observed-value audit (A3) accepted all 2,430 dataset pairs (100% coverage), including 100 with absolute Brier differences above 0.010. The risk-conditional audit (A4) rejected every pair, leaving 0% comparison coverage. Supporting analyses suggested sensitivity to training/evaluation policy alignment and little added value from response probes beyond inexpensive output comparisons. Matching missingness summaries therefore did not ensure matching prediction errors in this controlled experiment. Audit evaluation should report unresolved prediction differences together with the proportion of comparisons accepted. These descriptive development results lack external validation and do not establish clinical-harm thresholds.