When Coverage Lies: Uncovering Hidden Changes in Conformal Prediction Sets

Reusing an FP32-calibrated conformal threshold on a different numerical implementation can change individual prediction sets even when aggregate coverage changes little. We evaluate source-side input rankings for finding these disagreements with a limited number of target executions. The methods include threshold-margin ranking (PG-M), a weight-change sensitivity ranking (PG-R), confidence-based controls and a DFLARE-style input generator. The protocol was developed on 30 historical MLPs and evaluated once on fresh CIFAR-10 models: five seeds each of a small CNN and ResNet-18, across PyTorch and ONNX Runtime backends. With 100 target queries per model, PG-M found all 36 CUDA BF16 and all 7 FP16 disagreements of the small CNN, 129 of 136 INT8 disagreements, and all 75 INT8 disagreements of ResNet-18. The largest expected count under uniform selection was 5.4 (small CNN, INT8). For the probability score on the test set, mean coverage changed by at most 0.08 percentage points, while up to 565 prediction sets per backend changed, summed across five seeds at α = 0.1. The sensitivity term provided little additional benefit, and confidence-based controls were less effective at small query budgets. In a separate evaluation of explicit floating-point MLP programs, certificate screening reduced target calls by 95–99.5% but took 35–130 times longer than PG-M.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24374396
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When Coverage Lies: Uncovering Hidden Changes in Conformal Prediction Sets

S. M. Abdulla Al Mamun
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

When Coverage Lies: Uncovering Hidden Changes in Conformal Prediction Sets

S. M. Abdulla Al Mamun
preprint en

Abstract

Reusing an FP32-calibrated conformal threshold on a different numerical implementation can change individual prediction sets even when aggregate coverage changes little. We evaluate source-side input rankings for finding these disagreements with a limited number of target executions. The methods include threshold-margin ranking (PG-M), a weight-change sensitivity ranking (PG-R), confidence-based controls and a DFLARE-style input generator. The protocol was developed on 30 historical MLPs and evaluated once on fresh CIFAR-10 models: five seeds each of a small CNN and ResNet-18, across PyTorch and ONNX Runtime backends. With 100 target queries per model, PG-M found all 36 CUDA BF16 and all 7 FP16 disagreements of the small CNN, 129 of 136 INT8 disagreements, and all 75 INT8 disagreements of ResNet-18. The largest expected count under uniform selection was 5.4 (small CNN, INT8). For the probability score on the test set, mean coverage changed by at most 0.08 percentage points, while up to 565 prediction sets per backend changed, summed across five seeds at α = 0.1. The sensitivity term provided little additional benefit, and confidence-based controls were less effective at small query budgets. In a separate evaluation of explicit floating-point MLP programs, certificate screening reduced target calls by 95–99.5% but took 35–130 times longer than PG-M.

Zenodo (CERN European Organization for Nuclear Research)
University of Manchester (GB)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon