On the Cost of Conforming to Reviewers’ Expectations and the Potential Benefit of Prediction Competitions

Although academic peer review offers many important benefits, it can also impede scientific exploration. For instance, when reviewers share restrictive working assumptions, researchers may be incentivized to conform to them, even when alternative conjectures could better advance scientific understanding. We suggest mitigating this problem by complementing peer review with open prediction competitions. We illustrate the feasibility of this approach with two competitions on predicting decisions under risk. For more than 70 years, influential research on choice behavior has focused on models assuming that decision makers trust descriptions of payoff distributions, evaluate each option in isolation, and assign subjective values and weights to outcomes and probabilities. In contrast, the best-performing models in two recent prediction competitions relied on markedly different assumptions. They assume that decision makers question the accuracy of stated descriptions, use them as cues to retrieve similar past experiences, and select the strategy that proved most successful in analogous situations. We propose that, when evaluating papers that refine existing models, journal editors consider the decision “acceptance conditional on organizing an appropriate prediction competition.” Such competitions would test whether alternative assumptions achieve superior predictive performance, encouraging creativity and theoretical innovation while grounding scientific evaluation in predictive success.