Assessing the Reproducibility Probability of Tukey’s Honest Significant Difference Test via Nonparametric Predictive Inference Bootstrap

Reproducibility is a fundamental component of reliability in scientific research, reflecting the extent to which statistical conclusions remain stable under repeated experimentation. This paper investigates the reproducibility probability (RP) of Tukey’s Honest Significant Difference (HSD) test, a widely used post hoc procedure following one-way ANOVA, within the framework of nonparametric predictive inference (NPI), extending our earlier work on the reproducibility of the omnibus ANOVA decision to the pairwise post hoc decisions that follow it. RP is estimated via the NPI-bootstrap (NPI-B) across seven distributional scenarios (equal-variance normal, heteroscedastic normal, heavy-tailed, right-skewed (exponential and log-normal), bimodal, and discrete (Poisson) data) and sample sizes ranging from 10 to 100 per group, separately for the rejection and non-rejection regions defined by the significance threshold. Two patterns appear consistently across all scenarios. First, within the rejection region, RP for a fixed mean difference depends not only on the magnitude of the mean difference but on how far original p-values lie from the significance threshold: scenarios in which Tukey’s HSD attains high power even at small sample sizes (log-normal, Poisson, and bimodal data) achieve substantially higher RP than scenarios with more modest power (equal-variance normal, heteroscedastic, and heavy-tailed data), even for identical mean differences; RP is lowest and least sensitive to sample size under heavy-tailed data. Second, within the non-rejection region, RP decreases systematically as sample size increases, without exception, across every scenario and comparison examined, indicating that non-significant decisions obtained from larger samples are, in this sense, less reproducible than those obtained from smaller ones when a true effect is present. These findings show that RP captures aspects of decision stability not reflected in the p-value or statistical power alone, and its practical use is illustrated through an application to the PlantGrowth dataset.