Use as directed? A comparison of software tools intended to check rigor and transparency of published work
The causes of the reproducibility crisis include lack of standardization and transparency in scientific reporting. Checklists such as ARRIVE and CONSORT seek to improve transparency, but they are not always followed by authors and peer review often fails to identify missing items. To address these issues, there are several automated tools that have been designed to check different rigor criteria. We have conducted a broad comparison of 11 automated tools across 9 different rigor criteria from the ScreenIT group. We found some criteria, including detecting open data, where the combination of tools showed a clear winner, a tool which performed much better than other tools. In other cases, including detection of inclusion and exclusion criteria, the combination of tools exceeded the performance of any one tool. We also identified key areas where tool developers should focus their effort to make their tool maximally useful. We conclude with a set of insights and recommendations for stakeholders in the development of rigor and transparency detection tools. The code and data for the study is available at https://github.com/PeterEckmann1/tool-comparison.
Authors
- Lars Franzén
- Vladislav Nachev (ORCID: https://orcid.org/0000-0003-0521-6153)
- Martijn Roelandse (ORCID: https://orcid.org/0000-0001-5596-8084)
- Gerben ter Riet (ORCID: https://orcid.org/0000-0002-2231-7637)
- Maia Salholz‐Hillel (ORCID: https://orcid.org/0000-0003-1934-9504)
- Robert Schulz (ORCID: https://orcid.org/0000-0002-4830-309X)
- Peter Eckmann (ORCID: https://orcid.org/0000-0002-5388-9451)
- Sarah McCann
- Elisa Pilar Bascunan Atria
- Kaitlyn Hair
- Adrian Barnett
- Alexandra Bannach-Brown
- Cyril Labbe
- Małgorzata Anna Gazda
- Halil Kilicoglu
- Colby Vorland
- Tracey Weissgerber
- Anita Bandrowski
- Guillaume Cabanac
- James Howison
Institutions
- University of Illinois Urbana-Champaign (US)
- Queensland University of Technology (AU)
- Institut Universitaire de France (FR)
- Centre Hospitalier Universitaire de Toulouse (FR)
- University of California San Diego (US)
- Goudappel Coffeng (NL)
- Indiana University Bloomington (US)
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- University College London (GB)
- Amsterdam University of Applied Sciences (NL)
- Université de Montréal (CA)
- University of Coimbra (PT)
- The University of Texas at Austin (US)
- Université Grenoble Alpes (FR)
Publication Details
- Journal
- PLoS ONE
- Published
- 2026-02-13
- DOI
- https://doi.org/10.1371/journal.pone.0342225
- Primary Topic
- Meta-analysis and systematic reviews
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Silicon Valley Community Foundation
- Wellcome Trust
- European Commission
- Institut Universitaire de France
- National Institutes of Health
- U.S. National Library of Medicine