Bayesian Nonparametric Sensitivity Analysis of Multiple Test Procedures Under Dependence

This paper introduces a sensitivity analysis method for multiple testing procedures (MTPs) using marginal <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:semantics><mml:mi>p</mml:mi> <mml:annotation>$p$</mml:annotation></mml:semantics> </mml:math> -values. The method is based on the Dirichlet process (DP) prior distribution, specified to support the entire space of MTPs, where each MTP controls either the family-wise error rate (FWER) or the false discovery rate (FDR) under arbitrary dependence between <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:semantics><mml:mi>p</mml:mi> <mml:annotation>$p$</mml:annotation></mml:semantics> </mml:math> -values. This DP-MTP sensitivity analysis method provides uncertainty quantification for MTPs, by accounting for uncertainty in the selection of such MTPs and their respective threshold-based decisions regarding which number of smallest <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:semantics><mml:mi>p</mml:mi> <mml:annotation>$p$</mml:annotation></mml:semantics> </mml:math> -values are significant discoveries, from a given set of null hypothesis tested, while measuring each <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:semantics><mml:mi>p</mml:mi> <mml:annotation>$p$</mml:annotation></mml:semantics> </mml:math> -value's probability of significance over the DP prior predictive distribution of this space of all MTPs, and reducing the possible conservativeness of using only one such MTP for multiple testing. The DP-MTP sensitivity analysis method is illustrated through the analysis of over 28,000 <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:semantics><mml:mi>p</mml:mi> <mml:annotation>$p$</mml:annotation></mml:semantics> </mml:math> -values arising from hypothesis tests performed on a 2022 dataset of a representative sample of three million U.S. high school students observed on 239 variables. They include tests which, respectively, relate variables about the disruption caused by school closures during the COVID-19 pandemic, with various mathematical cognition, academic achievement, and student background variables. R software code for the DP-MTP sensitivity analysis method is provided in the Code and Data Supplement (CDS) of this paper.