Exact worst-case coverage of the jackknife+ and a 3/2 law for two-fold cross-validation conformal prediction

The jackknife+, CV+, and cross-conformal predictors build distribution-free prediction intervals without sacrificing data to a calibration split, but their assumption-free guarantees are essentially coverage 1 − 2α instead of 1 − α. Barber et al. (2021) asked whether the guarantee for K-fold CV+ can be sharpened. We determine the worst case in the two extreme regimes. For the jackknife+, the worst-case miscoverage over all distributions and all symmetric algorithms equals (2⌊α(n + 1)⌋ − 1)₊/(n + 1) for every n and every α ≤ 1/2. The matching construction uses i.i.d. data and an explicit algorithm built from a "zigzag" permutation that realizes a regular tournament through leave-two-out fits. For two-fold CV+ and cross-conformal prediction the factor 2 is not sharp: the worst-case miscoverage converges to (3/2)α as the fold size grows, with explicit non-asymptotic bounds. The reason is that the two fold p-values are asymptotically conditionally i.i.d. given the test point, and the average of two conditionally i.i.d. p-values needs only the factor 3/2. The same sharp inequality gives an exactly valid rule for averaging two independent random splits of split conformal prediction. For general K we give finite-sample bounds that improve both bounds of Barber et al. (2021). We also show that for every fixed K the large-fold worst case is at least α g_K(α), where g_K is the sharp averaging constant of K conditionally i.i.d. p-values. Since g_K → 2, the factor 2 returns as the number of folds grows. The exact worst case for fixed K ≥ 3 remains open. All results are proved, and exact computation is used only to check them.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-13
DOI
https://doi.org/10.5281/zenodo.23343499
Primary Topic
Statistical Methods and Inference
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Exact worst-case coverage of the jackknife+ and a 3/2 law for two-fold cross-validation conformal prediction

Hyunjun Yi
Zenodo (CERN European Organization for Nuclear Research)
Statistical Methods and Inference
preprint

Exact worst-case coverage of the jackknife+ and a 3/2 law for two-fold cross-validation conformal prediction

Hyunjun Yi
preprint en

Abstract

The jackknife+, CV+, and cross-conformal predictors build distribution-free prediction intervals without sacrificing data to a calibration split, but their assumption-free guarantees are essentially coverage 1 − 2α instead of 1 − α. Barber et al. (2021) asked whether the guarantee for K-fold CV+ can be sharpened. We determine the worst case in the two extreme regimes. For the jackknife+, the worst-case miscoverage over all distributions and all symmetric algorithms equals (2⌊α(n + 1)⌋ − 1)₊/(n + 1) for every n and every α ≤ 1/2. The matching construction uses i.i.d. data and an explicit algorithm built from a "zigzag" permutation that realizes a regular tournament through leave-two-out fits. For two-fold CV+ and cross-conformal prediction the factor 2 is not sharp: the worst-case miscoverage converges to (3/2)α as the fold size grows, with explicit non-asymptotic bounds. The reason is that the two fold p-values are asymptotically conditionally i.i.d. given the test point, and the average of two conditionally i.i.d. p-values needs only the factor 3/2. The same sharp inequality gives an exactly valid rule for averaging two independent random splits of split conformal prediction. For general K we give finite-sample bounds that improve both bounds of Barber et al. (2021). We also show that for every fixed K the large-fold worst case is at least α g_K(α), where g_K is the sharp averaging constant of K conditionally i.i.d. p-values. Since g_K → 2, the factor 2 returns as the number of folds grows. The exact worst case for fixed K ≥ 3 remains open. All results are proved, and exact computation is used only to check them.

Zenodo (CERN European Organization for Nuclear Research)
Statistical Methods and Inference
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.