Incentives, Assessment, and the Reliabiliity of Statistical Significance Assessments of Evidence

This analysis evaluates how researcher hypothesis selection incentives on the inferential value of empirical analyses. It illustrates the strength with which a “statistically significant” outcome objective incentivizes researcher aversion to testing possibly true null hypotheses in a comparatively unconstrained general setting. Left unfettered, it suggests that settings wherein researchers almost always opt to test false null hypotheses are common. That is, studies routinely produce reliable “falsifications” of a priori false hypotheses, a practice that transparently lacks inferential relevance. An outcome that aligns with the longstanding perspective that in many fields and settings the null hypotheses that researchers test are commonly false (e.g., Cohen, 1992). Collectively, the analysis illustrates the importance of comprehensive uncontrived understandings of researcher incentives and research assessment practices when evaluating the reliability and relevance of findings obtained from Null Hypothesis Significance Test based examinations of evidence.