Statistics in clinical urology research: fundamental concepts in comparative research

In this edition of the BJUI Trainees' Corner, we provide an essential summary of key statistical concepts in clinical urology research, offering valuable guidance for residents and fellows when writing scientific manuscripts. Drawing from the comprehensive ‘Guidelines for reporting of statistics for clinical research in urology’ [1], which were adopted by all four major urology journals, this commentary focuses on three fundamental principles from these guidelines when applied to comparative research, for example, when comparing the effects of two treatments on a clinically relevant endpoint in a randomised trial or an observational dataset. These principles are: (1) hypothesis testing, P values, and statistical significance; (2) interpretable estimates and clinical vs statistical significance; and (3) confidence intervals (CIs) and conclusions. These selected principles highlight common statistical misconceptions that can lead to inappropriate conclusions if not properly understood, ensuring that early-career researchers are well equipped to conduct and report their research with rigour and accuracy. For further reading, we recommend studying 3.1, 4.4, 4.5 and 6.3 in the Guidelines. Statistical analyses include two aspects: inference and estimation. Inference is the process of drawing conclusions from data. This is the foundation of hypothesis testing, where researchers analyse data to calculate P values. Inference can be thought of in terms of the question ‘Is something there?”, so for comparative research, inference addresses whether there is indeed a difference between groups. Estimation asks ‘How big is it?’ and so with respect to comparative research, involves calculating effect sizes, such as odds ratios, risk differences, or model coefficients, along with their CIs to quantify the magnitude of differences between groups. Just as a medical researcher would carefully consider biological mechanisms or relevant findings in existing studies, both facets of statistical inference must be thoughtfully addressed to draw appropriate conclusions from research data. Consider that we have performed a hypothesis test to detect a difference in UTI events after giving patients one antibiotic vs another and obtained our P value. Statistical significance is determined based on whether our P value is less than our alpha (0.05 is the value most widely used in medical research). If we find that the P value is < 0.05, our interpretation is straightforward: we can say, for example, ‘we found a statistically significant difference’ or that ‘we found sufficient statistical evidence of a difference’. Interpretation becomes tricky when our P value is not statistically significant (P ≥ 0.05). Often, investigators state that they found ‘no difference’. This is a serious misinterpretation of hypothesis testing: P values are the probability of observing a sample as extreme, or more than is currently observed, under the assumption that the null hypothesis is true for the population being studied. For example, assuming there is no difference in infection rates between two groups of patients who received different antibiotics, the P value is the chance we see a difference as extreme as observed or greater (Table 3). In other words, we consider the null hypothesis as true until we have sufficient evidence to reject it (P < 0.05). When the P value was ≥ 0.05, we did not have sufficient evidence to reject the null hypothesis or, in other words, we failed to reject the null hypothesis. The typical frequentist hypothesis testing framework does not set out to prove the null hypothesis but only to assess the level of evidence we have against it. Consequently, P values cannot be interpreted as levels of evidence for the truth of a null hypothesis. It is impossible to prove a negative, and the absence of evidence is not evidence of absence. The analogy in the Statistical Guidelines, rule 3.1 ‘Do not accept the null hypothesis’, is helpful here to conceptualise statistical significance. In a court case, you are assumed innocent until proven guilty, yet the jury gives a verdict of either ‘guilty’ (they found enough evidence for conviction) or ‘not guilty’ (they did not find enough evidence for conviction); there is no ‘innocent’ verdict. In addition to assuming that the null hypothesis is true, additional assumptions are imposed based on the specific statistical test being conducted. If assumptions are violated, the validity of the P value is in question. Further, the validity of statistical tests and P values are subject to other drawbacks of the study design including missing data and confounding. When interpreting the study, these considerations should be taken together. P values can be small because of large effect sizes or large samples. Consider a study comparing estimated blood loss between two surgical approaches, Surgery A and Surgery B. In Table 1, the mean difference in estimated blood loss is very small (6 mL), however, due to the large sample size (N = 1000 in each group), we have enough statistical power to detect this difference. Even in the case of P < 0.05, it is important to provide estimates of effect size (see Statistical Guidelines, rule 4.4 ‘Report estimates for the main study questions’). Although our results may be statistically significant, they do not imply clinical significance. This illustrates the need to evaluate clinical significance in addition to statistical significance, as a 6-mL difference in blood loss might be trivial in a clinical context. In scenarios with limited statistical power, it is also important to provide estimates of effect sizes along with estimates of variability such as CIs. As shown in Table 2, we do not have enough power to reject our null hypothesis (P = 0.14) but, clearly, the estimate of the difference in the percentage of successful urinary catheter placements by the expert vs the novice indicates that meaningful differences in the ability to place a difficult urinary catheter is consistent with our limited data. Reporting interpretable estimates of effect sizes is essential when drawing conclusions. Note that hazard ratios or odds ratios can be difficult to conceptualise; therefore, consider presenting risk differences or survival estimates at certain follow-up timepoints to give effect sizes that are more clinically interpretable. A CI provides a measure of the precision of our estimation and represents effect sizes most compatible with our data. The exact definition of a 95% CI is that if we were to take a large number of different samples and compute a 95% CI for each sample, then, on average, 95% of the CIs would contain the true effect size. So, the 95% confidence level refers only to how often 95% CIs computed from many studies would contain the true size if all assumptions used to compute the intervals were correct. Less formally, we can think of a 95% CI as the range of values that would not surprise us if the study were to be repeated. Note that in frequentist statistics, a CI excludes no value, no matter how far outside the interval. To make our conclusions, we need to determine whether the 95% CI for the difference between groups includes clinically relevant effect sizes. Clinically relevant effect sizes are with respect to, but not limited to, the outcome being measured. In Table 3, our CI does not include more than a 3.9% increase in UTIs associated with being in Treatment group A as compared to Treatment group B. We might say that, although the association was statistically significant, it was not clinically meaningful/significant based on the relative harms of the outcome vs the benefit of the treatment. However, if the outcome was death within 7 days, a 3.9% increase would likely be clinically meaningful/significant as well as statistically significant. For further reading, see Statistical Guidelines, rules 4.5 ‘Report CIs for the main estimates of interest’ and 6.3 ‘A statistically significant P value does not imply clinical significance’. Only when these concepts of statistical and clinical significance are taken together can we make appropriate conclusions from statistical inference. Table 4 shows examples of correct language to use when writing a scientific manuscript for combinations of different scenarios, assuming a well-designed study where the assumptions of the statistical approach are satisfied. Table 4 does not attempt to extend the language regarding specific treatment recommendations such as ‘treatment should be avoided/recommended’ as this will depend on a myriad of factors including study limitations, other findings in the literature, secondary outcomes, costs, etc. In summary, both components of statistical inference, hypothesis testing (inference) and estimation of effect sizes and CIs, play a critical role in accurately interpreting the results of a study. These elements work together to ensure that the conclusions drawn from research are both meaningful and reliable. For a more in-depth exploration of these concepts, we encourage readers to consult the ‘Guidelines for reporting of statistics for clinical research in urology’ [1], with particular attention to guidelines 3.1, 4.4, 4.5 and 6.3, as well as the section on the use and interpretation of P values. We hope that these guidelines serve as a valuable resource for researchers, helping to enhance the rigour and clarity of statistical reporting in clinical urology research. Both authors contributed equally to the concept, design, writing and editing of the manuscript and accepting the final version for publication. This work was supported in part by a National Institutes of Health/National Cancer Institute Cancer Center Support Grant (P30-CA008748) to Memorial Sloan Kettering Cancer Center. None.