Meta-analysis of Genuine and Fake p-Values

Abstract Sir Ronald Fisher, considering it important to consistently obtain significant p -values to support an experimental hypothesis, and thus replicating experiments to yield independent p -values, was one of the first to carry out a meta-analysis of p -values to obtain an overall decision. Suspecting also that Mendel only reported the results that closely agreed with the hereditary laws that Mendel wanted to establish, Fisher denounced the research malpractice, eventually scientific fraud, of researchers replicating experiments with the intent of reporting the best results, in his words “too good to be true”, to support their expectations or claims. On the other hand, publication bias, which largely results from published studies showing evidence of an effect, is a source of temptation to repeat experiments, especially when the first one carried out does not provide a significant p -value that favors its publication. Therefore, if the minimum of two independent “genuine” Uniform(0,1) p -values is reported to increase the chances of publication, it is a “fake” Beta(1,2) p -value, and its use in a meta-analysis changes the distributions of combined tests for genuine p -values. Mendel random variables, useful for modeling mixtures of genuine and fake p -values, are studied in detail, as well as their use in testing independence versus correlation of p -values. In addition, an example shows the usefulness of tables with critical values that consider the existence of a small proportion of fake p -values, to reach an overall informed decision when carrying out a meta-analysis of independent p -values with commonly used combined tests.