Auditing Conformal Prediction Sets Across Numerical Implementations

A split-conformal threshold calibrated with one numerical implementation of a classifier is often reused with another: a lower-precision weight copy, a different matrix-multiply mode or a different inference backend. We audit what that reuse does to prediction sets on 30 frozen multilayer perceptrons (Digits, MNIST, Fashion-MNIST; two architectures; five seeds) executed under 15 configurations, each derived independently from the original FP32 weights. A control first shows that an earlier version of our pipeline mutated weights in place: re-applying that mutation reproduces the historical CUDA FP32 outputs exactly for 30/30 models, so those historical GPU discrepancies were a pipeline artefact. In the corrected experiment, numerical changes still alter prediction sets while leaving top-1 predictions unchanged: with the probability score at α = 0.1, CPU BF16 changed 120 test-set prediction sets across the 30 model evaluations, 119 of them without an argmax change. Such changes are rare (largest group-mean rate 0.107%), and the largest absolute five-seed mean coverage difference is 0.102 percentage points. Finally, ranking unlabeled inputs by their source-side distance to the calibration threshold concentrates these rare disagreements under a deployment-query budget: at 100 queries it beat the exact random expectation in 113 of 420 model/runtime cases and tied in the remaining 307 zero-discovery cases, at the cost of scoring the whole source pool. The primary score and query budget were selected after earlier test results had been inspected. These exploratory results describe this benchmark.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24374562
Primary Topic
Adversarial Robustness in Machine Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Auditing Conformal Prediction Sets Across Numerical Implementations

S. M. Abdulla Al Mamun
Zenodo (CERN European Organization for Nuclear Research)
Adversarial Robustness in Machine Learning
preprint

Auditing Conformal Prediction Sets Across Numerical Implementations

S. M. Abdulla Al Mamun
preprint en

Abstract

A split-conformal threshold calibrated with one numerical implementation of a classifier is often reused with another: a lower-precision weight copy, a different matrix-multiply mode or a different inference backend. We audit what that reuse does to prediction sets on 30 frozen multilayer perceptrons (Digits, MNIST, Fashion-MNIST; two architectures; five seeds) executed under 15 configurations, each derived independently from the original FP32 weights. A control first shows that an earlier version of our pipeline mutated weights in place: re-applying that mutation reproduces the historical CUDA FP32 outputs exactly for 30/30 models, so those historical GPU discrepancies were a pipeline artefact. In the corrected experiment, numerical changes still alter prediction sets while leaving top-1 predictions unchanged: with the probability score at α = 0.1, CPU BF16 changed 120 test-set prediction sets across the 30 model evaluations, 119 of them without an argmax change. Such changes are rare (largest group-mean rate 0.107%), and the largest absolute five-seed mean coverage difference is 0.102 percentage points. Finally, ranking unlabeled inputs by their source-side distance to the calibration threshold concentrates these rare disagreements under a deployment-query budget: at 100 queries it beat the exact random expectation in 113 of 420 model/runtime cases and tied in the remaining 307 zero-discovery cases, at the cost of scoring the whole source pool. The primary score and query budget were selected after earlier test results had been inspected. These exploratory results describe this benchmark.

Zenodo (CERN European Organization for Nuclear Research)
University of Manchester (GB)
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon