An adaptive null proportion estimator for false discovery rate control

Abstract The false discovery rate is a commonly used criterion in multiple testing, and the Benjamini–Hochberg procedure is a standard approach to false discovery rate control. To increase its power, adaptive Benjamini–Hochberg procedures, that use estimates of the null proportion, have been proposed. A particularly popular approach being that based on Storey’s estimator. The performance of Storey’s estimator hinges on a critical hyperparameter, such that a pre-fixed configuration may lack power and existing data-driven hyperparameters may compromise false discovery rate control. In this work, we propose a novel class of adaptive hyperparameters and establish the false discovery rate control of the associated adaptive Benjamini–Hochberg procedure using a martingale argument. Within this class of data-driven hyperparameters, we further present a specific configuration designed to maximize the number of rejections and characterize its convergence to the optimal hyperparameter under a mixture model. The proposed method exhibits significant power gains, particularly in cases with a conservative null distribution, which are common in composite null testing, or with a moderate proportion of weak nonnulls, as is typically observed in biological experiments with enrichment processes.