Multiple Comparison Tests and Their Effects on Experimental Power

Animal scientists often need to compare more than two ways to raise their animals, necessitating multiple comparison tests. The most basic problem with multiple comparison tests is that they are based on an arbitrary threshold (p < 0.05) that separates significant from non-significant. This was important in the days before computers could analyze and compute actual probabilities from large quantities of raw data. Multiple test comparisons, as a whole, are not statistically independent (balanced or orthogonal) though some may be so). Scheffe’s and Fisher’s Least Significant Difference (LSD) tests closely approximated the reference paired t-test line, although not as well as orthogonal comparisons. The choice of experimental power is always based on considerations of how best to balance Type I and Type II errors. We show how some multiple comparison tests overestimate, some underestimate experimental power, and some over- and under-estimate in different parts of the curve. Only orthogonal comparisons closely follow reference-paired t-tests.