No Variance, No Verdict: Test-Retest Stability Gates for Repeated-Call LLM Experiments

Researchers frequently use preregistered test-retest correlation gates to ensure a large language model (LLM) has not drifted during data collection. This practice routinely fails when a model returns highly consistent responses near a ceiling or floor. Under boundary conditions, matched cells lack reliable variance, so the correlation measures binomial sampling noise rather than stable behavioural profiles, and perfectly stable models fail the gate. Correlation gates also contain a secondary blind spot. If a model's propensity shifts uniformly across all conditions, its rank order holds, and the gate retains a drifting system. Using an empirical study of frontier language models and systematic simulations, we demonstrate that standard correlation rules convert measurement limitations into incorrect exclusion decisions. To resolve this, we present a three-outcome screening framework that separates mean propensity shift from cell-specific movement. The screen tests overall run shift, evaluates whether the study design holds enough reliable between-cell information, and checks for local movement. By allowing a stability check to return a pass, fail, or indeterminate verdict, this framework requires prospective calibration and ensures the research design can support the stability decision.