Reporting of second-generation p-values, Bayes factors and fragility indices in trials enhances assessment of clinical importance beyond p-values: a meta-epidemiologic study

Frequentist methods of statistical inference in randomized controlled trials (RCTs) may inadequately reflect clinical importance of results. We assessed whether three complementary approaches validly assess clinical importance and their impact on original conclusions. Methods: In a sample of published superiority RCTs of health interventions, second-generation p-values, Bayes factors, and fragility indices were calculated for primary outcome results. Clinical importance was classified using effect size and confidence intervals relative to the reported minimum important difference. Agreement with clinical importance was measured by Cohen's Kappa estimates, positive predictive values (PPV) and negative predictive values (NPV). Results: In the sample of 269 trials, second-generation p-values, Bayes factors and reverse fragility indices increased with decreasing clinical importance, while fragility indices decreased with decreasing clinical importance. Second-generation p-values showed strong alignment with clinical importance derived from a common minimum important difference. Among statistically significant studies, Bayes factor and fragility index showed poor agreement but high (>96%) PPV and low (<25%) NPV, while in not statistically significant studies, Bayes factor failed to identify clinically important trials but identified all non-clinically important trials. Reverse fragility index showed moderate agreement and yielded a PPV of 94.1% and a NPV of 46.5%. Original trial conclusions changed in 31.6% of trials using second-generation p-values, in 20.9% using Bayes factors, and in 57% of trials using fragility indices. Discussion: Analysis plans and results interpretations should include quantitative assessments of clinical importance using second-generation p-values or Bayes factors, with robustness assessments using fragility indices.