The I2 Statistic As Selection Bias Test: Updated Threshold Limits

Aim: This study aims to revise the pre-specified I2 point estimate threshold limits of the trial-adjusted, simulated comparator trial (SCT)-based I2 test for selection bias in single randomised controlled trials (RCTs), in order to increase the percentage of testable RCTs. A further objective was to test the two null hypotheses: that, based on the test’s revised I2 point estimate thresholds, the magnitude of trial effect estimates is not significantly positively correlated with the selection bias levels (H01), and that it does not differ significantly between RCTs with identified ‘low’ and ‘high’ selection bias (H02). Methods: Revision of the I2 point estimate threshold limits was based on multiple RCT simulation and the re-testing of 332 real-world RCTs for selection bias. The selection bias level (B%) of each RCT, using the revised limits, was determined. H01 was tested using Spearman’s rank correlation, and H02 using the 2-tailed, independent samples t-test. The mean difference (MD) with 95% confidence interval (CI) between the mean absolute risk difference (RD) values with SD of RCTs with identified ‘low’ and ‘high’ selection bias was computed. Results: All 332 RCTs were testable, and none of the computed I2 point estimates fell outside the revised thresholds limits, increasing the proportion of testable RCTs from 71% to 100%. There was a statistically significant, positive, small correlation (0.1 ≤ |r| < 0.3) between the absolute RD values and established B% levels (Spearman’s rho = 0.268, p < 0.0001). The mean absolute RD for RCTs with identified ‘high’ selection bias was 0.18 (SD = 0.16), compared with 0.10 (SD = 0.13) for RCTs with ‘low’ selection bias. The difference was statistically significant: (t = -4.65; p < 0.0001; MD 0.08; 95%CI: 0.05 - 0.11). The effect size (Cohen's d = 0.52) indicates a medium effect. Both null hypotheses were rejected. Conclusion: The revised threshold limits increased the utility of the trial-adjusted, SCT based I2 test from 71 to 100% testable RCTs. The rejection of both null hypotheses indicates that the previously established statistically significant, positive relationship between selection bias levels (B%) and effect estimates (absolute RD) is independent of the changes in the I² threshold limits.