Selective Prediction Under Distribution Shift: The Failure of Threshold Transfer

Abstract A model can be allowed to say ``I do not know'' instead of giving an answer it may get wrong. This is called selective prediction. To make it work, someone picks a cut off value. If the model is less sure than that, it stays quiet. The cut off is picked on clean test data, so that the error rate among the answers it does give is at most some target, say 5%. This study shows the cut off does not survive when the input data changes. On CIFAR-10-C a cut off picked for 5% error gives 42% to 45% error at the worst corruption level. It gets worse when the target is stricter. At a 1% target the error is 19 times the promise. On CIFAR-10.1, a real new test set that only costs about 6 points of accuracy, the same cut off still doubles the promised error and stays quiet on nothing at all. The study then splits the system into two parts. One part sorts the inputs from most sure to least sure. The other part picks where to stop. Four different ways of sorting give almost the same result, within 1%. A perfect cut off reaches the target at every corruption level. So the sorting is fine. The stopping is broken. It lets in up to 77 extra points of data. Two published methods that guess accuracy without labels were tried as stopping rules. Both help, neither fixes it, and both lose to a very simple rule. A method with a proper statistical guarantee does fix it, but needs about 800 labelled examples from the new data under mild change and more than 4000 under strong change. The results hold for two model types, for four corruption types never used while building the study, and for real world change. Two versions of the guarantee method were built and thrown away, and both are reported with the reason.\\ Code Availability: The Jupyter notebooks implementing the experimental framework for selective prediction under distribution shift are publicly available at: https://github.com/shayamahmad/threshold-distribution_shift