Beyond <scp>PRISMA</scp> 2020 and <scp>AMSTAR</scp> 2: Further Actions Are Needed to Deal With Problematic Meta‐Analyses

For more than 10 years, researchers routinely complained that, because of the fast expansion of the scholarly literature, it was becoming very challenging for them to keep abreast of novel developments in even a very narrow portion of their discipline (e.g., Baveye 2014). At the same time, journal editors have experienced increasing difficulties to recruit reviewers (Siegel and Baveye 2010). Over the last few years, the situation does not appear to have improved significantly (West and Bergstrom 2021; Baveye 2021a, 2021b). The scholarly literature keeps expanding at an exponential rate. According to some estimates, 5.14 million articles were published during 2022, sizeably more than the 4.18 million published just 4 years earlier (Curcic 2023). More than ever, with conflicting demands on their time for teaching, supervising undergraduate and graduate students, reviewing for journals, or writing numerous proposals to compete for limited funding, researchers generally find it virtually impossible to devote as many hours as would be needed to read articles of direct interest to them in sufficient depth. Not surprisingly in this context, a significant effort has unfolded to review and synthesise relatively large bodies of literature and make their content more readily accessible to researchers and policy-makers. In recent years, tens of thousands of systematic reviews and especially of “meta-analyses” have been written. The staggering scale of the endeavour is evinced by the fact that the article of Page et al. (2021), proposing revised reporting guidelines for meta-analyses, has already been cited over 79,000 times, in only 3 years, according to Google Scholar (https://scholar.google.com; Last retrieved, January 29, 2025). Because it is proving time-consuming to stay abreast even of meta-analyses in virtually all disciplines, a trend is currently emerging of synthesising meta-analyses via what has been referred to as “second-order” meta-analyses (e.g., Schmidt and Oh 2013; Bergquist et al. 2023), or of carrying out “overviews of systematic reviews” (Lunny et al. 2024). In 2023 alone, more than 7000 articles referred to these practices, according to Google Scholar. No doubt part of the appeal of the meta-analysis method over the years has been its original description as a robust technique with a strong statistical foundation (Glass 1976; Shadish and Lecy 2015). Nevertheless, implementations of the method in practice have been the object of very strong criticisms, in particular in research on education (Abrami, Cohen, and d'Apollonia 1988; Ropovik, Adamkovic, and Greger 2021), medicine (Haidich 2010; Hedin et al. 2016; Chapman 2020), plant ecology (Koricheva and Gurevitch 2014), agronomy (Philibert, Loyce, and Makowski 2012; Beillouin, Ben-Ari, and Makowski 2019; Krupnik et al. 2019) and soil science (Fohrafellner et al. 2023), where researchers who have assessed the quality of meta-analyses found it overall to be low, and noticed that core quality criteria necessary to conduct sound meta-analyses did not appear to be well-understood by authors. Among the criteria that seem to be the most problematic in these various assessment exercises are the theory-based requirements for studies included in meta-analyses to be weighed according to the inverse of their variance, for meta-analyses to avoid mixing primary studies that have no connection with each other—the “apple and oranges” problem (Sharpe 1997)—, and for authors of meta-analyses to pay close attention to any bias that may exist in the literature, for example when journals solely publish articles describing positive, statistically-significant results. Mounting concern about the soundness of published meta-analyses and systematic reviews encouraged various groups to develop guidelines for either their rigorous implementation or proper reporting. Among them, the collaborative effort of QUORUM (Quality of Reporting of Meta-analyses), initiated in 1996, eventually led to the development of Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) (Moher et al. 2009), which was recently revised as PRISMA 2020 (Page et al. 2021). It appears to be adopted voluntarily by a growing number of researchers, and adherence to it is currently encouraged or even mandated by an increasing number of journals (Baveye 2024). Another set of guidelines is provided by AMSTAR (A MeaSurement Tool to Assess systematic Reviews), published in 2007 and revised in 2017 as AMSTAR 2 (Shea et al. 2007, 2017). In some ways, AMSTAR 2 provides a more comprehensive assessment of systematic reviews, since it focuses not only on their adherence to reporting standards but also on their methodological rigour, and addresses in detail issues such as protocol registration, risk of bias appraisals, and study selection processes. As with PRISMA 2020, some scholarly journals are now making the use of AMSTAR 2 mandatory for all new manuscripts (e.g., Chapman 2020). The current practice of adopting PRISMA 2020, AMSTAR 2, or other similar checklists, in the conduct and reporting of meta-analyses and systematic reviews is undoubtedly a step in the right direction. However, it raises two crucial questions. The first relates to whether this step will be sufficient, in and of itself, to weed out inadequate research syntheses in the future. There are indeed reasons to doubt whether, as both PRISMA 2020 and AMSTAR 2 assume implicitly, it is possible at this stage to diagnose accurately whether there is a significant publication bias in a given body of literature, whether something can be done to adjust for it, and, therefore, whether a meta-analysis or systematic review is meaningful. If this assumption is not warranted, the risk is that some meta-analyses that are currently published might turn out eventually, after additional research, to be fundamentally erroneous and misleading, which clearly would have devastating repercussions on the public's confidence toward science. The question therefore is whether something can be done relatively quickly to improve this situation. The second question relates to how we should handle research syntheses that were published in the past, did not follow checklists like PRISMA or AMSTAR, and are demonstrably deficient in one or more of their components. They are routinely being cited at present as entirely admissible evidence, when one might reasonably argue that they should not. The question is then how, practically, we can prevent citations to such potentially misleading publications. Following brief allusions by Baveye (2024) to these two questions and to how they can be dealt with by journal editors, we address them in more detail and from a broader perspective in the following. Over the years, it has become the implicit rule for scholarly journals in most disciplines not to publish articles describing ‘negative’ or non-statistically-significant results. Ritchie (2020) refers to this tradition as ‘one of science's most embarrassing secrets.’ It is so familiar to researchers that journal editors do not have to spell it out explicitly any more. Authors know that if they try to submit manuscripts that do not conform to the unwritten rule in this respect, it is almost guaranteed that they will not fare well in the review process, so that eventually they self-censure. In the least bad scenario, supposedly unpublishable results end up in the ‘grey’ (i.e., non-peer-reviewed) literature in the form of technical reports, unpublished Ph.D. dissertations, or papers in conference proceedings. Evidence suggests however that researchers often do not even bother writing up and sharing with colleagues results that are generally considered to be of negligible import, because they cannot lead to publications in peer-reviewed journals (Franco, Malhotra, and Simonovits 2014). Years ago, researchers realised that the reluctance of editors to publish negative or non-statistically-significant results meant that only a portion of all results would be available for review in meta-analyses, and therefore that the latter may be biased, leading to potentially erroneous conclusions (e.g., Eysenck 1984, 1994; Rothstein 2008; Haidich 2010; Lin and Chu 2018; Furuya-Kanamori, Barendregt, and Doi 2018; Nair 2019; Ritchie 2020; Borenstein et al. 2021). Different methods have been devised to assess whether a significant publication bias exists in a given body of literature. Unfortunately, most of these methods have been shown time and again to have shortcomings, so that after several decades a consensus still does not appear to exist among specialists about the best assessment approach (e.g., Rothstein 2008; Fragkos, Tsagris, and Frangos 2014; McShane, Böckenholt, and Hansen 2016; Lin and Chu 2018; Shi and Lin 2019; Mathur and VanderWeele 2020; Afonso et al. 2024; Lunny et al. 2024). One way out of this conundrum involves trying to convince statisticians to keep developing novel, more sophisticated methods to assess if a given body of literature exhibits publication bias. This is clearly a daunting challenge, since assessing publication bias implies basically trying to say something conclusive about a situation in which a portion of the relevant data is entirely missing and unknown. In this respect, Ritchie (2020) refers derisively to techniques like the commonly used ‘funnel’ plots to detect publication bias (Egger et al. 1997), and the thorough analysis by Shi and Lin (2019) of the “trim and fill” method, which relies on filling the data gaps, shows that it is fraught with difficulties. Nevertheless, research in this area needs to be encouraged, new approaches need to be explored, and precise information needs to be obtained on the conditions under which publication bias can be effectively detected and adjusted, so that meta-analyses and systematic reviews become feasible. Based on attempts to deal with the publication bias issue over the last 20 years, one could argue that the only full-proof way to deal with it is to avoid it in the first place. As Rothstein (2008) wrote already 16 years ago, ‘Over the long run, the best way to deal with publication bias is to prevent it’. That means practically that scholarly journals should start routinely publishing what is currently considered to be ‘negative’ results. Some journals have tried to convey the idea that they would be open to this approach. The editors of the journal PLOS One, for example, state on its website that they ‘evaluate research on scientific validity, strong methodology, and high ethical standards—not perceived significance’, which appears to head in the right direction. However, this attitude is neither widely shared, nor adopted by the high Impact Factor journals at this point. What it would take to make progress rapidly in that respect would be for journal editors to let it be known, clearly and unambiguously, to prospective authors that their journals would welcome manuscripts regardless of the direction, positive or negative, of the effects they observed, or their level of statistical significance, as long as the research was carried out with an appropriate level of rigour (Baveye 2024). Editors who would be willing to follow this path could find support for it in the work of the philosopher Karl Popper (1963), who, in a nutshell, stated that theories cannot be validated, but only invalidated or falsified. From that perspective, which many researchers have espoused over the years, one could argue that the rigorous, carefully carried out experiments that are the most valuable for the advancement of science are those that clearly invalidate a given hypothesis, or at least show that differences predicted to be significant on the basis of it under specific circumstances turn out not to be so in practice. In editorials in which they would describe steps implemented by their journal to reduce the incidence of publication bias, editors could perhaps invoke this Popperian viewpoint to rationalise the shift in their policy. Some advocates for the status quo may consider that this new editorial policy would amount to a drastic change in our publishing culture, which researchers would find difficult to adjust to or may suspect to generate articles of low quality. However, one might argue that it would not necessarily be such a major revolution in many cases, and would only require a minor adjustment of how we set up working hypotheses to structure reports on our work. Especially in disciplines like mine (soil science) in which we deal with very complex systems, many aspects of which we still do not understand, much of the research that we carry out is exploratory in nature, driven by questions but not necessarily by definite hypotheses. In spite of that, when writing the ensuing manuscripts, we often feel we have to formulate a hypothesis a posteriori, to which the experiments are then supposed to lend support. Under these conditions, without any loss of rigour or scientific integrity, nothing would prevent us from setting up working hypotheses at the write-up stage in a way that the experiments appear to produce a positive outcome, to circumvent any reluctance of the part of journal editors or reviewers toward publishing negative results. The soil science article of Wu et al. (2010) illustrates this approach. It is very likely that it would never have been accepted for publication if the working hypothesis had stated that a particular measurement technique (visible–near infrared spectroscopy) can directly detect heavy metals in soils, and the results had turned out negative. However, the article stipulated instead, based on prior knowledge, that the spectroscopic method should theoretically not be expected to ‘see’ heavy metals in soils, and the article ended up with a confirmation of this working hypothesis, an apparently positive message that was very well received by reviewers. Whenever it is suggested that editors of scholarly journals should allow the publication of ‘negative’ or non-statistically-significant results, it is common for someone to what of structure would be needed for that to more than 16 years after Rothstein (2008) out how crucial that as editors and researchers are it is not that would be As the situation currently both groups would a and their work would become more if research syntheses that are being published in increasing would turn out eventually to be or entirely erroneous and That alone, one would by the loss of that would likely from the public's and in research, should make it for editors and researchers to try to the and prevent the publication of problematic meta-analyses and systematic For the same should be in the publication of all of the research they regardless of the of the and could a significant in that the who to the from the advancement of for example, the of research to develop new or some of their on scientific should also be in making that scientific publications be as as and, in that research syntheses be based on of results of just very The what should be done to as much as possible that meta-analyses published in the appropriate quality However, this still open the question of how one should handle meta-analyses that were published in the past, did not one or more quality may be misleading, and therefore, one could should not be a large this issue meta-analyses and is for all manuscripts, not just those articles that have been by their authors or by the journals in which they were published to be cited years as if they were still In where quality have shown some meta-analyses to be (e.g., Haidich 2010; and Gurevitch 2014; Loyce, and Makowski 2012; et al. 2019; Krupnik et al. 2019; et al. 2023), their number is that reviewers could be to them in manuscripts for and to either that they not be or that be by However, the number of meta-analyses in that is to the many tens of thousands of meta-analyses in the literature as a Editors cannot already reviewers to assess meta-analysis that is cited in manuscripts they are to and to weed out Another approach is As suggested by Baveye one could a of of articles that were either or in the peer-reviewed literature to be in some and this to reviewers. that such an approach would be one needs to carefully at how such a should be set what the most way would be for technical editors to it, and what information from it should be to reviewers. the way to these questions without technical is to them, that describe first the of information that would be to reviewers and the in which it should be about how this information could be Whenever reviewers are to assess manuscripts citations to meta-analyses, they need to know whether these citations are that whether they fare well on checklists like PRISMA 2020, AMSTAR 2, or similar AMSTAR 2 can be in this context, as it could as an additional to studies that did not to methodological a systematic way to these for For each cited regardless of the checklists that are an overall could be for example, et al. However, such a may not be since it might be by among At the other just the for each of the quality criteria in the checklists might even if a is provided et al. 2024; et al. 2024). might of out a few criteria the of primary data with the inverse of their and that may be as would generate the of cited meta-analyses provided to The most approach would be for technical editors to use that would the of manuscripts, use if they published meta-analyses in more than a way (e.g., as part of the publications on a given but not in of their assess the of the cited meta-analyses by checklists like PRISMA 2020 or AMSTAR 2, and issue a the in the to reviewers. The could be as new in the to deal with the problem of publication bias. possible to technical editors being to all manuscripts to this is that it would their However, one could argue that the would not be very than what they are already routinely at present to for In this as they use which the very rapidly to if of or in the manuscripts are from other the does not take much time, the results may in some require some The use of a would in a number of be very it might be and for two One is that of this might be up by information about assessed meta-analyses in a that the would to any new That the would become more and more the more it is and therefore the time needed to citations in a given would become The other the of such a of would a for technical editors than the is that reports on meta-analyses should be to who would then whether or not there is a problem out to the authors. develop the needed to published meta-analyses, a effort initiated a can as a up with a to detect statistical in peer-reviewed articles by articles for statistical results, the in the and the two to if they et al. that of the articles they at least one that was with its a in the 2016; 2018; Ritchie 2020). this more than articles have been with alone, more than articles to eventually on more than of At this a growing number of journals are as part of their The used by and other similar that have been in the based on could relatively be used for the development a to meta-analyses in the literature. one could find as well in the of Shi and Lin who primary data from meta-analyses in to find out to what the and method could be to adjust for publication bias. tens of thousands of and meta-analyses published each and about the quality of some of these meta-analyses been in the past, the research the risk of to deal in the with many published research articles erroneous misleading This would likely have a devastating on the public's confidence in science and its to in many (e.g., Ritchie 2020; Baveye 2024). The and by many journals, of comprehensive reporting and implementation guidelines for meta-analyses and systematic reviews (e.g., PRISMA 2020 and AMSTAR a positive step in this context, and may in the effectively authors from meta-analyses that do not Unfortunately, the use of these guidelines still two questions both PRISMA 2020 and AMSTAR 2 assume that it is possible at this to diagnose accurately the of publication bias in a given which is a to carry out any of research research is needed in this In the the only in this would be for scholarly journals to start routinely publishing ‘negative’ or non-statistically-significant results. This might seem like a drastic from a long but for various it does not have to be so if one is willing to set up working hypotheses when writing up manuscripts, or if one that the of theories is crucial to the advancement of science. the current use of checklists like PRISMA 2020 or AMSTAR 2, it will undoubtedly make the conclusions of meta-analyses more raises the question of the appropriate way to deal with the many thousands of meta-analyses and systematic reviews that have been published in the at a time when the use of these checklists was not and to be cited in this respect might be the development of a which technical editors of scholarly journals could use to information to reviewers on the soundness of the meta-analyses cited in the manuscripts they are to with both of these of we could to a a of years from when with problematic meta-analyses would be or even entirely Baveye is the of the up with the that is wrote the and its after The that the research was in the of any or of any that could be as of in the broader of the in Baveye sharing not to this article as no were or during the current