Decision value of confidence intervals versus significance testing in surgical research: a mixed methods study

The misuse and misinterpretations of statistical analyses in medical research have been discussed for several decades and many authors have suggested this has led to poor clinical decision-making. Significance testing, the conventional statistical approach in medical research, has been suggested to be the leading cause of the misunderstandings. Despite the extensive discussion in the literature, there is limited evidence on whether the problems in medical research are linked to the lack of usefulness of significance testing with clinicians’ decision-making process. The purpose of this study was to investigate what statistical information in published research articles surgeons found valuable for their clinical decision-making. The aim was to build a validated instrument that could effectively describe the findings. The findings suggest surgeons favor information that supports statistical prediction and refining surgical skills. Personal advice from credible surgeons and technique articles were rated similarly in usefulness to meta-analyses and prospective cohort studies. The term “significance” was found to be a noisy term and had a subjective meaning based on the reader’s interpretation. Significance testing and confidence intervals were rated to have the same value. When comparing the informal definition of confidence intervals to p-values, confidence intervals were rated more useful than p-values. A modified formula of expected utility theory has been presented to account for a skill’s conditional effect on a decision’s outcomes.