Configuration-Invariant Conformal Prediction for Large Language Models under Dynamic Inference Shifts

Modern large language models (LLMs) are typically accessed as black-box services whose outputs are produced under a configuration chosen at inference time: sampling temperature, system prompt, and (for chat models) format constraints. Different operators choose different configurations, and the same operator may change the configuration after a calibration set has been collected. This dynamic configuration shift violates the exchangeability assumption underlying standard split conformal prediction (CP), causing prediction sets whose coverage and size deteriorate silently. We make the failure explicit on a controlled 2x2 shift grid (in-distribution, temperature shift, prompt shift, joint shift), where off-the-shelf baselines (SSCP, APS, RAPS) lose up to 0.6 in empirical coverage. We then propose Configuration-Invariant CP (Inv-APS), a distribution-free method that (i) rescales the pre-softmax logit vector so its dispersion matches a calibration-time reference, and (ii) builds prediction sets from a cumulative-mass nonconformity score with an entropy-aware per-example threshold. Both ingredients are computed from returned token probabilities alone and require no model retraining, no access to logits, and no knowledge of the test-time configuration. Across the four shift configurations Inv-APS recovers empirical coverage in [0.88, 0.92] (target 0.90) while producing sets that are 30-50% smaller than APS and RAPS.