My System Is As Effective As Yours: Reproducibility, Sustainability, and More

When a paired or two-sample t-test shows that a difference is not statistically significant (i.e., the null hypothesis H is "accepted"), all we obtain as a "conclusion" is: " we cannot conclude from the data whether the difference is real or not. " In general, statistical significance tests with a significance level of α give us a proper conclusion with 100(1-α)% confidence only if the H is rejected. Hence, for example, in a reproducibility study in IR, traditional significance tests like the t-test are not adequate if one wishes to claim that the reproduced run is as good as the target run. In this paper, we show that the equivalence test, which has been gaining popularity in the medical domain for comparing new drugs with standard drugs, is applicable to IR research in situations such as above. We also discuss the less ambitious noninferiority test, which can be used, for example, when an IR researcher wants to claim that their system performs no worse than a state-of-the-art system that is computationally much more expensive and therefore earth-unfriendly. The equivalence and noninferiority tests require researchers to pre-define an indifference zone, which defines how much difference is "practically negligible"; hence we also discuss one possible approach to establishing an indifference zone for a given IR effectiveness measure. In addition, we provide recommendations for IR researchers who wish to conduct equivalence/noninferiority tests involving more than two systems. It is hoped that equivalence and noninferiority tests will become standard practices in IR, so that we can avoid making claims based on lack of statistical significance.