Conformal Prediction under Open-Set Label Shift: Identification of the Novel Proportion and Robustness to Class Imbalance and Label Noise

Conformal prediction guarantees that a prediction set contains the test label with a prescribed probability when calibration and test data are exchangeable. Under open-set label shift, the test distribution contains classes that never appear in the calibration data, and this guarantee no longer applies. CP-POL addresses this setting with prediction sets over the observed labels and a calibrated novelty flag based on softmax confidence, assuming balanced classes, clean labels and a well-trained classifier. We study which of its guarantees survive when these assumptions fail and when the softmax confidence is replaced by an arbitrary novelty score. Coverage of observed labels and control of the false-positive rate of the flag hold for any score and any classifier. For the proportion of novel classes, we show that the bound obtained from a score, the proportion identified by the law of the score, and the canonical proportion of semi-supervised novelty detection form a chain of inequalities, with explicit conditions for equality. Under label shift among observed classes, the pooled bound on the novel proportion can be positive when no novel class is present; we give a bound that remains valid under arbitrary shift. Under contamination of the calibration sample by novel points, all guarantees are preserved under a tail-dominance condition and degrade by explicit amounts otherwise. Simulations and a held-out-class study on handwritten digits illustrate each result.