Exact prevalence regions: finite-sample uncertainty quantification, threshold decisions, and monitoring for class-prevalence analytics under prior probability shift

Abstract Many analytics decisions consume class prevalences rather than individual labels: the market share of a sentiment, the case burden of a disease, the drift state of a deployed model. Quantification methods estimate the prevalence vector of an unlabeled batch under prior probability shift, but they return point estimates, and their uncertainty companions, resting on large-sample approximations, resampling, or a prior, carry no finite-sample frequentist guarantee. I introduce exact prevalence regions (EPR): confidence regions obtained by inverting exact Monte Carlo two-sample tests against reference samples drawn from held-out, class-conditional calibration pools. Because prior probability shift makes each null hypothesis exactly simulable, the region is finite-sample valid for any classifier, any permutation-symmetric statistic, and any sample size, marginally over calibration sampling; the exchangeability the test uses is derived from the sampling assumptions, and the cost of its failure under within-batch dependence is measured. The same construction yields exact one-sided tests for threshold decisions and a day-over-day monitoring workflow whose representability alarm flags batches that leave the label-shift family. Across 18 datasets and more than 300,000 evaluation bags, against audited bootstrap, delta-method, and Bayesian baselines, EPR held its nominal level wherever the shift assumption held by construction, including 0.950 on all 5000 official bags of the 28-class LeQua 2022 task, where bootstrap coverage spanned 0.030–0.974 across implementations and seeds. The cost is width, 1.13–1.43 times the delta method’s outside rare-class regimes, and reduced power in threshold decisions; both are quantified and distilled into practitioner guidance.

Authors

Institutions

Publication Details

Journal
International Journal of Data Science and Analytics
Published
2026-11-12
DOI
https://doi.org/10.1007/s41060-026-01359-4
Primary Topic
Imbalanced Data Classification Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Exact prevalence regions: finite-sample uncertainty quantification, threshold decisions, and monitoring for class-prevalence analytics under prior probability shift

Seyed Armin Hosseini
International Journal of Data Science and Analytics
Imbalanced Data Classification Techniques
article

Exact prevalence regions: finite-sample uncertainty quantification, threshold decisions, and monitoring for class-prevalence analytics under prior probability shift

Seyed Armin Hosseini
article en

Abstract

Abstract Many analytics decisions consume class prevalences rather than individual labels: the market share of a sentiment, the case burden of a disease, the drift state of a deployed model. Quantification methods estimate the prevalence vector of an unlabeled batch under prior probability shift, but they return point estimates, and their uncertainty companions, resting on large-sample approximations, resampling, or a prior, carry no finite-sample frequentist guarantee. I introduce exact prevalence regions (EPR): confidence regions obtained by inverting exact Monte Carlo two-sample tests against reference samples drawn from held-out, class-conditional calibration pools. Because prior probability shift makes each null hypothesis exactly simulable, the region is finite-sample valid for any classifier, any permutation-symmetric statistic, and any sample size, marginally over calibration sampling; the exchangeability the test uses is derived from the sampling assumptions, and the cost of its failure under within-batch dependence is measured. The same construction yields exact one-sided tests for threshold decisions and a day-over-day monitoring workflow whose representability alarm flags batches that leave the label-shift family. Across 18 datasets and more than 300,000 evaluation bags, against audited bootstrap, delta-method, and Bayesian baselines, EPR held its nominal level wherever the shift assumption held by construction, including 0.950 on all 5000 official bags of the 28-class LeQua 2022 task, where bootstrap coverage spanned 0.030–0.974 across implementations and seeds. The cost is width, 1.13–1.43 times the delta method’s outside rare-class regimes, and reduced power in threshold decisions; both are quantified and distilled into practitioner guidance.

International Journal of Data Science and AnalyticsVol. 22(1)
University of Sussex (GB)
Imbalanced Data Classification Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon