Network of uncertainties: Network meta‐analyses often does not mention key methodological components

Network meta-analysis (NMA) is a methodological approach that extends beyond the traditional pairwise meta-analysis by its ability to compare directly and indirectly interventions and rank them in accordance with the effect size observed.1 First described in the literature by Lumley in 2002, this method enables multiples comparisons simultaneously,2, 3 but as an emerging technique, it encompasses considerable methodological heterogeneity, as standardized recommendations for its execution and reporting are still evolving. Nevertheless, reporting and methodological guidelines have been established that outline essential components necessary for a robust NMA.4, 5 To construct a valid and trustworthy NMA, it is imperative to guarantee that key methodological assumptions are fulfilled, including transitivity, coherence and consistency.5 The goal of this study is to critically appraise the NMAs published in PubMed Central (PMC), with focus on the methodological aspects and choices adopted by their authors, since when carefully planned and conducted, NMAs are valuable instruments to support evidence-informed decision-making. ‘Coherence’ or ‘Incoherence’ ‘Transitivity’ or ‘Intransitivity’ ‘Consistency’ or ‘Inconsistency’ ‘Rank’ or ‘Rankogram’ ‘GRADE’ or ‘CINeMA’ Subsequently, the references retrieved were screened by two independent reviewers who manually checked the inclusion criteria, and any conflicts were solved through consensus between all authors. Only full-text articles were included. Commentaries, letters to the editor, corrections, retracted articles, and ‘abstract only’ retrieved texts were excluded. The final list of included and excluded articles is presented in the Supporting Information Material: S2. A summary of the selection process is depicted in Figure 1. All statistical analyses and database merging procedures were performed using Stata 18, and the code used were made available in the supporting Information Material: S2. From the 1826 sources retrieved for screening, 1765 met the inclusion criteria and were incorporated into the analysis following a manual selection conducted by two independent and blinded reviewers, with any conflicts resolved through consensus among all authors. The objective of the manual selection was to confirm the references were indeed reporting a systematic review with network meta-analysis. Of the 1765 included sources, 30.9% (n = 545) mentioned the term ‘Transitivity’ or ‘Intransitivity’ at least once. Additionally, 14% (n = 247) included the term ‘Coherence’ or ‘Incoherence’. The terms ‘Consistency’ or ‘Inconsistency’ were found in 86.2% (n = 1521) of the sources. Furthermore, 48.7% (n = 860) used the terms ‘GRADE’ or ‘CINeMA’. Lastly, 65.6% (n = 1157) referenced the term ‘Rank’ or ‘Rankogram’. These terms were searched throughout the entirety of each text (Table 1). The findings of this appraisal underscore a significant degree of variability in the reporting of key methodological components among NMAs reported in PubMed Central. Despite the critical importance of consistency, coherence, transitivity, certainty of evidence, and ranking in ensuring the robustness and reliability of NMAs, the extent to which these elements are reported varies considerably across studies. Transitivity was mentioned in 30.9% (545/1,765) of the analyzed sources. This is concerning given that transitivity is a fundamental assumption in NMA, ensuring that indirect comparisons are valid.6 Transitivity extends the concept of clinical and methodological homogeneity to comparisons across multiple studies evaluating different treatments. It requires that when intervention A is compared in studies against both intervention B and intervention C, the characteristics of intervention A (including any effect modifiers) remain consistent across these comparisons. This ensures that the relative effects observed between A and B and between A and C are reliable and can be meaningfully interpreted.5 The relatively low reporting rate of transitivity suggests a potential oversight in addressing this key methodological aspect, which could compromise the validity of the synthesized evidence. Transitivity is statistically represented through consistency (or coherence), which occurs when the relationships implied by the data align with the transitivity assumption. In other words, consistency (or coherence) is observed when the differences between treatments in indirect comparisons are supported by the data, affirming the validity of transitivity across the network of studies.5 Coherence was addressed in 14% (247/1765) of the studies. In contrast, consistency was reported in 86.2% (1521/1765) of the sources, indicating a relatively high level of adherence to this guideline. The high reporting rate suggests that researchers recognize the importance of evaluating and maintaining consistency in their analyses. However, it is possible that researchers are using the term ‘consistency’ as a synonym for transitivity. This potential conflation could lead to misunderstandings and misapplications of these distinct concepts, further emphasizing the need for precise and consistent terminology in NMA reporting. Certainty of evidence, as assessed by the report of the terms ‘GRADE’ or ‘CINeMA,’ was reported in 48.7% (860/1765) of the studies. This low level of reporting highlights an area for improvement, as evaluating the certainty of evidence is critical for informing clinical decision-making and policy formulation. We found that 65.6% (1157/1765) of systematic reviews with network meta-analyses ranked the assessed interventions evaluated. However, only 49.3% (570/1,157) of the studies that mentioned ranking also evaluated the certainty of evidence, highlighting a gap in comprehensive reporting. This discrepancy suggests that while many researchers recognize the importance of ranking interventions, fewer incorporate a thorough evaluation of the reliability of the evidence supporting these rankings, which is essential for making informed clinical and policy decisions. Ensuring that ranking is accompanied by assessment of the certainty of evidence is crucial for providing a complete picture of the comparative effectiveness of interventions.5 This appraisal of NMAs in PubMed Central reveals significant gaps in the reporting of key methodological elements, particularly transitivity, which are crucial for the validity of indirect comparisons and the overall reliability of the analysis. While consistency is commonly mentioned, the incomplete linkage between ranking and certainty assessment indicates areas needing improvement. Additionally, the potential conflation of consistency or coherence with transitivity highlights the need for clearer and more precise terminology. The methods employed in this study were limited to retrieving only studies that provide full-text articles and were available in PubMed Central. Furthermore, our assessment was based on whether the authors reported any of the specified terms within any part of the text available in PubMed Central. Consequently, if any of the searched terms were present exclusively in the supplementary material, our search methodology would not have detected them. It is important to highlight that we did not judge, for example, if the included study assessed transitivity. We only report if they used the word transitivity or intransitivity in their manuscript. Given that ranking is extensively utilized by policymakers and decision-makers in healthcare to determine the appropriateness and implementation of interventions, we strongly recommend that rankings be accompanied by a thorough assessment of the certainty of evidence using strategies such as GRADE or CINeMA. This practice will ensure that the rankings are not only informative but also reliable and actionable, thereby enhancing the overall trustworthiness and applicability of NMA findings. Additionally, there needs to be a consensus on the use of the terms ‘incoherence’ and ‘inconsistency.’ Currently, these terms are often used interchangeably, which can be redundant. Clarifying and standardizing the terminology will improve the clarity and precision of reporting and interpretation in NMA. To advance the field of NMA and enhance the reliability and clinical utility of its findings, it is essential to promote adherence to established methodological guidelines. Increasing awareness and training on the importance of transitivity, coherence, and certainty of evidence, as well as ensuring comprehensive reporting of these elements, will be critical steps. Strengthening these areas will ultimately lead to more trustworthy and actionable insights, thereby improving clinical practice and policymaking. Victor Alexandre dos Santos Valsecchi: Writing—original draft. Lara Faria Souza Dias: Writing—original draft. Rachel Riera: Conceptualization; supervision; project administration. Rafael Leite Pacheco: Conceptualization; writing—original draft; writing—review and editing; project administration. All authors were given final approval. The authors declare no conflict of interest. All data and codes are available at: http://github.com/rlpacheco/network_meta_analyses_methodological_components.git. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.