Peer Review and the Technical Soundness of Research: Evidence from Automated Audits of Political Science Working Papers
Peer review is meant to select high quality work and make it better. To measure one dimension of quality (“technical soundness”), we develop method-specific checklists and a “universal checklist” derived from the published literature by a transparent, rule-based procedure. We then have a language-model coder apply these checklists to 798 working papers posted from 2020 to 2022 on three preprint servers and to 406corresponding published versions. A one standard deviation increase in preprint soundness is associated with a 10.3 percentage point increase in publication probability and higher journal placement. For the published/unpublished margin, both the universal and the method-specific checklists predict publication outcomes to a similar degree. A “naive” review by the same model, with no checklist, predicts publication aboutas well overall and less well for top-three journals, but the checklists catch issues this misses and add further predictive power. Published versions score modestly higher than preprints: 67% score higher at publication, and the mean overall score rises by +0.19 standard deviations. We release the checklists and the auditing system as a tool for authors.
Authors
- Andrew T. Little (ORCID: https://orcid.org/0000-0001-9911-4291)
- Tak‐Huen Chau (ORCID: https://orcid.org/0000-0001-7622-1000)
Publication Details
- Journal
- SocArXiv (OSF Preprints)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.31235/osf.io/zk26u_v1
- Primary Topic
- Academic Publishing and Open Access
- Type
- preprint