Peer Review and the Technical Soundness of Research: Evidence from Automated Audits of Political Science Working Papers

Peer review is meant to select high quality work and make it better. To measure one dimension of quality (“technical soundness”), we develop method-specific checklists and a “universal checklist” derived from the published literature by a transparent, rule-based procedure. We then have a language-model coder apply these checklists to 798 working papers posted from 2020 to 2022 on three preprint servers and to 406corresponding published versions. A one standard deviation increase in preprint soundness is associated with a 10.3 percentage point increase in publication probability and higher journal placement. For the published/unpublished margin, both the universal and the method-specific checklists predict publication outcomes to a similar degree. A “naive” review by the same model, with no checklist, predicts publication aboutas well overall and less well for top-three journals, but the checklists catch issues this misses and add further predictive power. Published versions score modestly higher than preprints: 67% score higher at publication, and the mean overall score rises by +0.19 standard deviations. We release the checklists and the auditing system as a tool for authors.

Authors

Publication Details

Journal
SocArXiv (OSF Preprints)
Published
2026-11-18
DOI
https://doi.org/10.31235/osf.io/zk26u
Primary Topic
Academic Publishing and Open Access
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Peer Review and the Technical Soundness of Research: Evidence from Automated Audits of Political Science Working Papers

Andrew T. Little, Tak‐Huen Chau
1 links
SocArXiv (OSF Preprints)
Academic Publishing and Open Access
preprint

Peer Review and the Technical Soundness of Research: Evidence from Automated Audits of Political Science Working Papers

Andrew T. Little, Tak‐Huen Chau
preprint en

Abstract

Peer review is meant to select high quality work and make it better. To measure one dimension of quality (“technical soundness”), we develop method-specific checklists and a “universal checklist” derived from the published literature by a transparent, rule-based procedure. We then have a language-model coder apply these checklists to 798 working papers posted from 2020 to 2022 on three preprint servers and to 406corresponding published versions. A one standard deviation increase in preprint soundness is associated with a 10.3 percentage point increase in publication probability and higher journal placement. For the published/unpublished margin, both the universal and the method-specific checklists predict publication outcomes to a similar degree. A “naive” review by the same model, with no checklist, predicts publication aboutas well overall and less well for top-three journals, but the checklists catch issues this misses and add further predictive power. Published versions score modestly higher than preprints: 67% score higher at publication, and the mean overall score rises by +0.19 standard deviations. We release the checklists and the auditing system as a tool for authors.

SocArXiv (OSF Preprints)
Academic Publishing and Open Access
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon