Estimating the historical minimum false‐positive risk of statistically significant reported outcomes in anaesthesia and pain medicine

Clinical research is driven by the search for safe and efficient interventions to improve patient outcomes. Traditionally, treatment efficiency is assessed within the framework of null hypothesis significance testing. However, this does not address a crucial question when reporting a statistically significant result: what is the risk that my discovery (in the sense of statistical significance) is wrong [1]? Equating this false-positive risk with the p value would correspond to one of many misinterpretations of statistical testing [2]. To illustrate this problem, I will statistically estimate the minimum false-positive risk in a large set of reported outcomes in randomised controlled trials in anaesthesia and pain medicine. Reported outcomes were selected from the publicly available data set by van Zwet et al. [3] containing the published results of more than 68,000 studies from the Cochrane Database of Systematic Reviews. Here, only studies from the domain ‘Anaesthesia and Pain’ were selected, and only efficacy-related outcomes from randomised controlled trials were analysed. Two-sided p values were computed from reported test statistics (z-values). The false-positive risk for each statistically significant outcome was calculated as follows [4]: FPR 1 − FPR = BF 01 · Pr H 0 Pr H 1 $$ \frac{FPR}{1- FPR}={BF}_{01}\cdotp \frac{\Pr \left({H}_0\right)}{\Pr \left({H}_1\right)} $$ , where BF01 denotes the Bayes factor ( BF 01 = Pr data H 0 Pr data H 1 $$ {BF}_{01}=\frac{\Pr \left( data|{H}_0\right)}{\Pr \left( data|{H}_1\right)} $$ ). The minimum Bayes factor was calculated using the ‘-e P log(P)’ calibration [5], which assumes p values to be distributed uniformly between 0 and 1 under the null hypothesis and smaller p values to be more likely under the alternative hypothesis. I computed the minimum false-positive risk for the entire range of pretest probabilities regarding the null or alternative hypothesis being true. This pretest probability could refer to the long-term proportion of truly effective treatments evaluated in clinical trials from a frequentist perspective or the prior probability investigators assign to the efficacy of an intervention before running a trial. All computations were performed with R (version 4.4.2, R Foundation, Vienna, Austria). In total, 8707 reported outcomes from 2570 studies were analysed (Table 1). The distribution of the observed test statistics is shown in Figure 1a. The median (IQR) minimum false-positive risk of all statistically significant outcomes was 5% (0–32%) (assuming an equal probability of the null and alternative hypothesis being true). Figure 1b provides a more detailed perspective on the minimum false-positive risk with respect to the magnitude of the p values and prior beliefs. The minimum risk of falsely declaring a treatment effect when there is none is small (e.g. < 5%) for outcomes most incompatible with the null hypothesis (p < 0.005). Figure 1b also shows that, for reported outcomes with p values between 0.005 and 0.05, a strong prior belief for the existence of an effect – or a very large proportion of truly effective interventions that are being tested in the long run – is required to reach a minimum false-positive risk < 5%. The importance of the false-positive risk in interpreting the results of clinical trials is being increasingly appreciated. An analysis of the false-positive risk in more recent trials in the journal Anaesthesiology showed a median (IQR) false-positive risk of 6% (1–22%) when assuming equal pretest probabilities [6]. Similarly, the key finding of this analysis is that about half of the statistically significant reported outcomes feature a minimum false-positive risk > 5% and about one in four outcomes feature a risk of at least 18% of rejecting the null hypothesis falsely (in the case of equal pretest probabilities). The analysis presented here thus further strengthens previous notions that a considerable portion of findings based on statistical significance possibly declared the presence of a treatment effect when there is likely no effect [6]. As shown in Figure 1b, this is of particular importance in the case of barely statistically significant outcomes: for those 32% of the outcomes, the median minimum risk of falsely declaring the existence of a treatment effect is 23% and can reach up to 32%. These results are thus a point in case that the risk of false-positive statistically significant findings is large when focusing on only a moderate level of significance (i.e. the commonly selected threshold of 5%) [7]. A high false-positive risk can be further expected in the presence of underpowered studies [7]. The median power of the trials compiled by van Zwet et al. is only 13% [3] and low power is also likely an issue in the set of trials considered here. Given that reported outcomes in anaesthesia and pain medicine feature one of the largest proportions of significant treatment effects in the Cochrane Database of Systematic Reviews and larger small-study effects compared with many other domains [8], clinicians are encouraged to consider the false-positive risk in the interpretation of clinical trials. I thank Professor Frank Stüber and Professor Dominik Günsch for supporting the research activities and Professor Patrick Wüthrich and Thomas Vetsch for valuable inputs on the manuscript. Statistical code is available at https://github.com/marbhuber/fpr_anaesthesiology_pain and outcome data is available from https://osf.io/xjv9g/overview. No external funding or competing interests declared. Open access publishing facilitated by Inselspital Universitatsspital Bern, as part of the Wiley - Inselspital Universitatsspital Bern agreement via the Consortium Of Swiss Academic Libraries.