Toward reliable false discovery rate control in classification problems under distribution shift

BACKGROUND: Domain shifts or batch effects can significantly degrade the performance of machine learning-based classifiers in software production without any warning. This can particularly be problematic in medical applications which involve human lives. The shift changes the shape of the prediction score distribution of the test samples compared to that of the training samples; hence, decision boundary re-calibration would be required to conform to the expected performance standards. METHOD AND FINDINGS: We present a simple, yet robust heuristic method to calculate valid p-values (uniformly distributed) for raw prediction scores of test samples under data shift in general classification problems. Our method is based on an adjustment of the test prediction scores so that the test null distribution (approximated distribution of the prediction scores of negative test samples) aligns to the training null distribution (actual distribution of prediction scores of the negative training samples). We termed it the Test Null Adjustment (TNA) method. Then, empirical p-values of the adjusted test prediction scores can be calculated with respect to the training null distribution to be used in any statistical applications which require test samples accompanied by valid p-values. We demonstrate the usability of our TNA method in FDR control with Benjamini-Hochberg (BH) protocol. CONCLUSION: The main advantage of the TNA method is that its application does not require any domain specific knowledge. It works in binary and multi-label classification scenarios, it is fully data-driven, and it operates in the 1-dimensional space of the prediction scores ensuring stable numerical calculations. We have performed four experimental tests on the biomedical image analysis domain to demonstrate that the TNA method is able to mitigate the bias in FDR induced by data shift in real-life applications. TNA is available at https://github.com/Aborevsky01/TNA.

Authors

Institutions

Publication Details

Journal
PLoS ONE
Published
2026-11-16
DOI
https://doi.org/10.1371/journal.pone.0343253
Primary Topic
Domain Adaptation and Few-Shot Learning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Toward reliable false discovery rate control in classification problems under distribution shift

Attila Kertész‐Farkas, Andrey Borevskiy
1 links
PLoS ONE
Domain Adaptation and Few-Shot Learning
article

Toward reliable false discovery rate control in classification problems under distribution shift

Attila Kertész‐Farkas, Andrey Borevskiy
article en

Abstract

BACKGROUND: Domain shifts or batch effects can significantly degrade the performance of machine learning-based classifiers in software production without any warning. This can particularly be problematic in medical applications which involve human lives. The shift changes the shape of the prediction score distribution of the test samples compared to that of the training samples; hence, decision boundary re-calibration would be required to conform to the expected performance standards. METHOD AND FINDINGS: We present a simple, yet robust heuristic method to calculate valid p-values (uniformly distributed) for raw prediction scores of test samples under data shift in general classification problems. Our method is based on an adjustment of the test prediction scores so that the test null distribution (approximated distribution of the prediction scores of negative test samples) aligns to the training null distribution (actual distribution of prediction scores of the negative training samples). We termed it the Test Null Adjustment (TNA) method. Then, empirical p-values of the adjusted test prediction scores can be calculated with respect to the training null distribution to be used in any statistical applications which require test samples accompanied by valid p-values. We demonstrate the usability of our TNA method in FDR control with Benjamini-Hochberg (BH) protocol. CONCLUSION: The main advantage of the TNA method is that its application does not require any domain specific knowledge. It works in binary and multi-label classification scenarios, it is fully data-driven, and it operates in the 1-dimensional space of the prediction scores ensuring stable numerical calculations. We have performed four experimental tests on the biomedical image analysis domain to demonstrate that the TNA method is able to mitigate the bias in FDR induced by data shift in real-life applications. TNA is available at https://github.com/Aborevsky01/TNA.

PLoS ONEVol. 21(11)
National Research University Higher School of Economics (RU), Centre of Advanced Studies (RU)
Domain Adaptation and Few-Shot Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon