Is the latest message still the best forecast? A pre-registered re-evaluation of machine learning for satellite collision risk

Is the latest conjunction data message (CDM) still the best forecast of a satellite encounter's final collision risk? The European Space Agency's 2019 Collision Avoidance Challenge asked whether machine learning could beat the naive forecast that the latest risk will hold, two days before closest approach; only 12 of 97 teams matched or beat it. We re-evaluate the challenge in four pre-registered rounds. On the official loss, the answer is yes, as far as the benchmark can tell: on the sealed test, no model beat the naive forecast with confidence (best: −4.7%, 97.5% confidence interval [−14.0, +3.6]%), the organizers' significance test does not survive correction for overlapping resamples (p = 0.36), and a simulated team with no information reaches rank 10 of 97 from leaderboard feedback alone. As a probability forecast and a false-alarm filter, the answer is no. In cross-validation, our model forecasts whether an encounter ends high-risk with 10% lower log loss than a calibrated latest risk, and, in a post-hoc check with thresholds set inside the training folds, removes 24% of the naive forecast's false alarms while keeping 96% of its true calls. The risk in the messages is consistent with a coherent forecast near the high-risk line, but encounters below 10⁻⁸ later reach 10⁻⁶ at 16 times the rate a coherent probability allows (p ≤ 4.5 × 10⁻⁴ even counting missions instead of encounters). On the official loss, one more day of messages lowers the naive forecast's loss by 45% at the operational class mix, while the model adds nothing measurable at two days.