Non‐significant does not mean uninformative – but poor reporting can be misleading

Two publications in this issue of BJUI address a problem that is rarely considered: how urological randomised controlled trials (RCTs) with non-significant results are reported and interpreted [1, 2]. Together, they discuss errors in statistical reasoning and offer a framework for extracting more meaningful inference from trials that do not meet conventional thresholds for statistical significance. Randomised trials in urology often produce statistically non-significant, ‘negative’ or ‘null’, results. This is merely a consequence of studying clinically realistic interventions producing modest effects among heterogeneous populations. The problem arises when we report and interpret those results. The Sharifan research letter [1] provides a reminder that interpretative errors remain common in contemporary urology trials. By reviewing recent RCTs, the author documents frequent acceptance of the null hypothesis and persistent use of ‘trend’ language for borderline P values. These practices represent logical errors that exaggerate certainty and obscure what the data show. This journal and other leading urology journals have jointly published explicit statistical reporting guidelines stating that null hypotheses are rejected or not rejected, never accepted, and that P values just above conventional thresholds should not be described as trends [3]. These principles reflect the logic of frequentist inference. That these issues persist despite clear guidance and statistical review suggests that the problem lies with entrenched habits of interpretation and ‘autopilot’ reporting of results without regard for their underlying meaning. The persistence of these issues has a long history. More than three decades ago, Oxford biostatistician Douglas Altman [4] highlighted the prevalence of statistical misinterpretation in clinical research. Subsequent audits of the medical literature confirmed that statistical errors were widespread, even in high-impact journals. The Sharifan [1] analysis suggests that while reporting standards have improved, core misunderstandings about inference remain remarkably durable. The letter by Sharifan [1] focuses on how non-significant results are misrepresented. The review by Noll et al. [2] addresses a more fundamental question: how should such results be interpreted at all? The authors apply likelihood ratios to urology RCTs with non-significant primary outcomes, asking whether the observed data provide greater support for the null hypothesis or for a clinically meaningful alternative. This distinction matters. A P > 0.05 tells us only that the data are not sufficiently incompatible with the null hypothesis. It does not tell us whether a study was inconclusive or whether it provides evidence that a clinically important effect is unlikely. Likelihood ratios, while not a substitute for thoughtful trial design, force that distinction into the open. The authors demonstrate that some ‘negative’ trials provide little information either way and may justify further investigation, whereas others provide strong evidence against benefit. Under conventional hypothesis testing, both scenarios are typically labelled the same: non-significant. This is a limitation of the inferential framework, not of the trials themselves. Similar Bayesian re-analyses in other surgical disciplines have reached analogous conclusions [5]. This finding has practical consequences. Some ‘negative’ trials should prompt larger studies or meta-analyses. Others should prompt us to stop investigating the problem. P values alone cannot tell us which is which. A broader point remains regarding the incentivisation to produce manuscripts with P < 0.05 (selective reporting). This is a known and continuing issue with considerable implications for the reproducibility of research [6]. As editors and reviewers, we frequently encounter submitted manuscripts that clearly prioritise statistical significance over clinical significance; or observational studies where the number of P values in the paper are many multiples of the sample size, with those P < 0.05 (un)helpfully adorned with bolding and asterisks. As our BJUI reporting guidelines recommend [3], we instead encourage authors to focus on estimation rather than inference, that is report your most important mean difference, percentage change, hazard ratio etc., and its CI, rather than a forest of P values. A manuscript that asks an interesting question, has a study design to answer it, collects robust data and analyses that data appropriately is 90% of the way to being accepted for publication in peer-reviewed journals. The solution is not to abandon P values, but to interpret them cautiously, use them judiciously and in conjunction with CIs —and, where appropriate, to supplement with other approaches that contribute to answering the clinical questions we actually care about. An artificial intelligence–based language model (ChatGPT, OpenAI) was used to assist with drafting and language editing of this editorial. The authors are solely responsible for the content, interpretation, and conclusions. The authors have no other disclosures pertaining to this submission.