Quantile and threshold calibration separate even for Gaussian forecasts

Unrefereed research candidate. In one sentence: two standard ways of checking whether probabilistic forecasts are calibrated, quantile checks and threshold checks, can disagree even for smooth Gaussian forecasts, so passing one does not certify the other; this settles a conjecture of Gneiting and Resin (2023) and locates the exact boundary for outcomes with finitely many ordered values. The question. A probabilistic forecast is calibrated if its stated probabilities can be taken at face value. A threshold check fixes an outcome value x and asks whether, among forecasts that give the same probability F(x) of staying below x, that probability matches the observed frequency. A quantile (or conditional exceedance probability) check fixes a level α and asks whether, among forecasts with the same α-quantile, the outcome falls below it a fraction α of the time. For a smooth forecast, “F(x) = α” and “the α-quantile is x” pick out the same forecasts. Gneiting and Resin (2023, Theorem 2.16) proved the two notions equivalent when the forecast takes countably many values and conjectured that countability could be dropped; Resin, Yang and Gneiting (2026) still draw these implications as conjectured arrows. Results. (1) Countability cannot be dropped. Gaussian forecasts N(M, S2) with M uniform on [−1, 1] and S uniform on [1, 2], paired with explicit smooth outcome laws whose densities stay within a factor of two of the forecast densities, separate the notions in both directions: one law satisfies conditional exceedance and quantile calibration with an exactly uniform PIT, yet fails threshold and marginal calibration at every threshold; the other is threshold calibrated but fails quantile calibration, with a PIT distribution function strictly below the uniform one at every interior level. The errors reach about one percentage point: for example the outcome falls below 0 with probability 0.50758 while the average forecast probability is 1/2, which a binomial test at the 5% level with 80% power detects with about 34,000 cases. (2) For outcomes with k ordered values and positive forecast probabilities, conditional exceedance calibration implies threshold calibration when the forecast has countable support (any k) or k ≤ 3 (any forecast law), and fails for k ≥ 4 with an uncountable forecast family. (3) For nominal outcomes, full probabilistic calibration implies marginal calibration, with the sharp total-variation bound 2d − d2; class-wise calibration does not imply average probabilistic calibration (an example listed by Resin, Yang and Gneiting already indicates this; it is verified here, and an example with strictly positive probabilities is given). The nominal implications fCC⇒fTC and pCC⇒pTC hold for k ≤ 3 or countable support and remain open for four or more classes. (4) In location–scale families every compact perturbation preserving either property is described, with a general change-of-conditioning identity, transfer bounds and a local construction in any smooth forecast family. Why. Both checks condition on the same set of forecasts, a line in parameter space of probability zero, but weight it differently: the quantile check uniformly, the threshold check in proportion to the forecast's scale. An error that averages to 0 under one weighting averages to 1/9 (times the perturbation size) under the other: the Borel–Kolmogorov phenomenon of conditioning on zero-probability events. Limits. Population constructions, not a study of any deployed forecast system. The perturbation size 1/10 rests on a computer-assisted interval-arithmetic bound; the analytic proof gives 1/400. Two nominal implications remain open for four or more classes. Finite-sample test theory is not treated. Contents. The paper (LaTeX source and PDF) with its figure and figure script; three Python verification scripts (symbolic and numerical checks of the Gaussian examples and effect sizes; exact rational checks of the discrete and nominal results; an interval-arithmetic certificate of the amplitude bound) with recorded outputs; the bibliography from registry records; a machine-readable index; a research-gate record; a replay receipt and SHA-256 sums. Assurance. Written proofs, one computer-assisted interval certificate, and replayed checks. Internal AI-assisted review only (five reviewer roles, an adversarial review and audit by a second AI model, clean replays, one revision round, a re-review and a proof audit of new proofs): not peer review, independent reproduction or formal verification. Creator: Anonymous. Text and data CC0-1.0; code MIT.

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24259221
Primary Topic
Forecasting Techniques and Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Quantile and threshold calibration separate even for Gaussian forecasts

Zenodo (CERN European Organization for Nuclear Research)
Forecasting Techniques and Applications
preprint

Quantile and threshold calibration separate even for Gaussian forecasts

preprint en

Abstract

Unrefereed research candidate. In one sentence: two standard ways of checking whether probabilistic forecasts are calibrated, quantile checks and threshold checks, can disagree even for smooth Gaussian forecasts, so passing one does not certify the other; this settles a conjecture of Gneiting and Resin (2023) and locates the exact boundary for outcomes with finitely many ordered values. The question. A probabilistic forecast is calibrated if its stated probabilities can be taken at face value. A threshold check fixes an outcome value x and asks whether, among forecasts that give the same probability F(x) of staying below x, that probability matches the observed frequency. A quantile (or conditional exceedance probability) check fixes a level α and asks whether, among forecasts with the same α-quantile, the outcome falls below it a fraction α of the time. For a smooth forecast, “F(x) = α” and “the α-quantile is x” pick out the same forecasts. Gneiting and Resin (2023, Theorem 2.16) proved the two notions equivalent when the forecast takes countably many values and conjectured that countability could be dropped; Resin, Yang and Gneiting (2026) still draw these implications as conjectured arrows. Results. (1) Countability cannot be dropped. Gaussian forecasts N(M, S2) with M uniform on [−1, 1] and S uniform on [1, 2], paired with explicit smooth outcome laws whose densities stay within a factor of two of the forecast densities, separate the notions in both directions: one law satisfies conditional exceedance and quantile calibration with an exactly uniform PIT, yet fails threshold and marginal calibration at every threshold; the other is threshold calibrated but fails quantile calibration, with a PIT distribution function strictly below the uniform one at every interior level. The errors reach about one percentage point: for example the outcome falls below 0 with probability 0.50758 while the average forecast probability is 1/2, which a binomial test at the 5% level with 80% power detects with about 34,000 cases. (2) For outcomes with k ordered values and positive forecast probabilities, conditional exceedance calibration implies threshold calibration when the forecast has countable support (any k) or k ≤ 3 (any forecast law), and fails for k ≥ 4 with an uncountable forecast family. (3) For nominal outcomes, full probabilistic calibration implies marginal calibration, with the sharp total-variation bound 2d − d2; class-wise calibration does not imply average probabilistic calibration (an example listed by Resin, Yang and Gneiting already indicates this; it is verified here, and an example with strictly positive probabilities is given). The nominal implications fCC⇒fTC and pCC⇒pTC hold for k ≤ 3 or countable support and remain open for four or more classes. (4) In location–scale families every compact perturbation preserving either property is described, with a general change-of-conditioning identity, transfer bounds and a local construction in any smooth forecast family. Why. Both checks condition on the same set of forecasts, a line in parameter space of probability zero, but weight it differently: the quantile check uniformly, the threshold check in proportion to the forecast's scale. An error that averages to 0 under one weighting averages to 1/9 (times the perturbation size) under the other: the Borel–Kolmogorov phenomenon of conditioning on zero-probability events. Limits. Population constructions, not a study of any deployed forecast system. The perturbation size 1/10 rests on a computer-assisted interval-arithmetic bound; the analytic proof gives 1/400. Two nominal implications remain open for four or more classes. Finite-sample test theory is not treated. Contents. The paper (LaTeX source and PDF) with its figure and figure script; three Python verification scripts (symbolic and numerical checks of the Gaussian examples and effect sizes; exact rational checks of the discrete and nominal results; an interval-arithmetic certificate of the amplitude bound) with recorded outputs; the bibliography from registry records; a machine-readable index; a research-gate record; a replay receipt and SHA-256 sums. Assurance. Written proofs, one computer-assisted interval certificate, and replayed checks. Internal AI-assisted review only (five reviewer roles, an adversarial review and audit by a second AI model, clean replays, one revision round, a re-review and a proof audit of new proofs): not peer review, independent reproduction or formal verification. Creator: Anonymous. Text and data CC0-1.0; code MIT.

Zenodo (CERN European Organization for Nuclear Research)
Forecasting Techniques and Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon