MATES: A tool for appraising the completeness with which a meta-analysis has been reported

Meta-analysis is commonly a core component of systematic reviews and has become an important method to reconcile conflicting findings, increase statistical power, and chart new research directions. However, poor reporting practices make it challenging to evaluate the validity of meta-analyses. Despite the existence of reporting checklists, a specifically designed tool has yet to be developed to appraise the completeness with which a meta-analysis has been reported. To bridge this gap, we introduce the Meta-analysis Appraisal Tool for Environmental Sciences (MATES). To develop MATES, we adapted a Delphi process involving experts in meta-analysis methodologies, researchers with experience in guideline/appraisal tool development, and editors of relevant journals. The Delphi process had five steps, including three workshops (11-16 participants), a survey (193 participants), and a validation task (30 participants). This iterative development process resulted in a 14-item appraisal tool that reflects the environmental science and research syntheses community's consensus on essential elements to appraise the completeness with which a meta-analysis has been reported. Validation across 50 meta-analyses demonstrated that the tool is repeatable (average agreement rate: 88.97 %) and time-efficient to implement (17.00 ± 12.23 min). We also outline guidance for interpreting MATES results, describe its potential applications, and reflect on the development process. The authors provide practical implementation guidance for each MATES item, illustrated with real examples in the supplementary material. We also report an extended development methodology to support reproducibility. Finally, we built created a ShinyApp that includes both a training module and an application tool to enhance the usability of MATES (https://kylemorrisonisshiny99.shinyapps.io/MATES_shiny/). Overall, MATES provides authors, readers, stakeholders, and editors with a reliable and accessible tool for appraising the completeness with which a meta-analysis in environmental sciences has been reported.

Authors

Institutions

Publication Details

Journal
Environment International
Published
2025-11-29
DOI
https://doi.org/10.1016/j.envint.2025.109935
Primary Topic
Meta-analysis and systematic reviews
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

MATES: A tool for appraising the completeness with which a meta-analysis has been reported

Malgorzata Lagisz, Jing Gan, Kyle Morrison, Damien Beillouin et al.
Environment International
Meta-analysis and systematic reviews
article

MATES: A tool for appraising the completeness with which a meta-analysis has been reported

Malgorzata Lagisz, Jing Gan, Kyle Morrison, Damien Beillouin, Eliza Grames, Patrice Pottier, Yefeng Yang, Paul Whaley, Ayumi Mizuno, Verónica Ferreira, Ziqian Xia, April Robin Martinig, Coralie Williams, Pietro Pollo, Alexandro B. Leverkus, Guang Hao, Mojtaba Keikha, Shinichi Nakagawa, Lorenzo Ricolfi, Zachary Simpson, Evangelia Ntzani, David Moher, Rose E. O’Dea, Silvina Magdalena Manrique, Mona Ghannad, Francesco Latterini, Emily A. Hennessy, Erin L. Macartney, Simone Jaqueline Cardoso, Joanna IntHout, Shanika Nanayakkara, Matthew J. Page, Brian Gallagher, Kok Sin Woon, Betina Kozlowsky-Suzuki, Dakis-Yaoba Ouédraogo, Harriet Sleight, Edward Pursell, T M Kiran Kumara
article en

Abstract

Meta-analysis is commonly a core component of systematic reviews and has become an important method to reconcile conflicting findings, increase statistical power, and chart new research directions. However, poor reporting practices make it challenging to evaluate the validity of meta-analyses. Despite the existence of reporting checklists, a specifically designed tool has yet to be developed to appraise the completeness with which a meta-analysis has been reported. To bridge this gap, we introduce the Meta-analysis Appraisal Tool for Environmental Sciences (MATES). To develop MATES, we adapted a Delphi process involving experts in meta-analysis methodologies, researchers with experience in guideline/appraisal tool development, and editors of relevant journals. The Delphi process had five steps, including three workshops (11-16 participants), a survey (193 participants), and a validation task (30 participants). This iterative development process resulted in a 14-item appraisal tool that reflects the environmental science and research syntheses community's consensus on essential elements to appraise the completeness with which a meta-analysis has been reported. Validation across 50 meta-analyses demonstrated that the tool is repeatable (average agreement rate: 88.97 %) and time-efficient to implement (17.00 ± 12.23 min). We also outline guidance for interpreting MATES results, describe its potential applications, and reflect on the development process. The authors provide practical implementation guidance for each MATES item, illustrated with real examples in the supplementary material. We also report an extended development methodology to support reproducibility. Finally, we built created a ShinyApp that includes both a training module and an application tool to enhance the usability of MATES (https://kylemorrisonisshiny99.shinyapps.io/MATES_shiny/). Overall, MATES provides authors, readers, stakeholders, and editors with a reliable and accessible tool for appraising the completeness with which a meta-analysis in environmental sciences has been reported.

Environment InternationalVol. 207
Universidade Federal de Juiz de Fora (BR), Australian National University (AU), Binghamton University (US), The University of Sydney (AU), Centre de Coopération Internationale en Recherche Agronomique pour le Développement (FR), United States Department of Agriculture (US), Radboud University Nijmegen (NL), University of Ottawa (CA), University of Alberta (CA), The University of Melbourne (AU), Universidad de Granada (ES), Indian Council of Agricultural Research (IN), Universidad de Alcalá (ES), Guangdong Pharmaceutical University (CN), University of Ioannina (GR), Université de Montpellier (FR), Kerman University of Medical Sciences (IR), Ottawa Hospital (CA), Sydney Dental Hospital (AU), Radboud University Medical Center (NL), Westmead Hospital (AU), UNSW Sydney (AU), Massachusetts General Hospital (US), Manpower Employment Organisation (GR), Instituto Andaluz de Ciencias de la Tierra (ES), Institute of Agricultural Economics (BG), Institute of Dendrology (PL), Radboud Institute for Molecular Life Sciences (NL), Monash Health (AU), Forests and Societies (FR), Xiamen University Malaysia (MY), Ottawa Public Health (CA), BC Children's Hospital (CA), Center for Systems Biology (US), Environmental Earth Sciences (AU), Anglia Ruskin University (GB), University of York (GB), Monash University (AU), Concordia University (CA), Lancaster University (GB), University of Coimbra (PT), Okanagan University College (CA), Universidade Federal do Estado do Rio de Janeiro (BR), University of the Philippines Diliman (PH), University of Hong Kong (HK), Stanford University (US)
University of New South Wales, Canada Excellence Research Chairs, Government of Canada
Openalex Percentile: Top 64%
Meta-analysis and systematic reviews
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.