Cost-Aware Conformal Learning for Reliable Accident Detection Under Exchangeability Violations

Conformal prediction can give an accident detector coverage guarantees on its confidence, but only when the calibration and test data are exchangeable. Accident image benchmarks break it: a hash audit of three benchmarks finds 9% to 59% of test images have an exact copy in training, confirmed by learned embeddings and structural similarity. The damage falls on confidence, not accuracy: after grouping, accuracy stays near 98%, yet a detector calibrated on leaky data appears to hold a 90% guarantee while covering only 67% of novel images, a gap of 22.1 points with the calibration sets matched. We build a duplicate grouped split and report on a locked test set, where empirical coverage is near 90%. A cost-aware allocation of the coverage budget, our main contribution, is model-agnostic: it raises accident coverage from 91% to 98% and lowers accidents declared normal for every base method in every run, at the price of normal coverage falling to 80%, false alerts rising from seven to 33, and 9% human review, so the 90% budget is a frequency-weighted average, not a per-class level. On a rare severe class, 0.4% of records, it cuts missed cases from 34 to three.