Investigating the reproducibility of the social and behavioural sciences

Published claims should be reproducible, yielding the same result when applying the same analysis to the same data. We assessed reproducibility in a stratified random sample of 600 papers published from 2009 to 2018 in 62 journals spanning the social and behavioural sciences. Authors of 144 (24.0% [95% CI 20.8 - 27.6%]) papers made data available to assess reproducibility, and for 38 others, we obtained source data to reconstruct the dataset. We assessed 143 of the 182 available datasets and found that 76.6 (53.6% [95% CI 45.8 - 60.7%]) papers were rated as precisely reproducible and 105.0 (73.5% [95% CI 66.4 - 80.0%]) papers as at least approximately reproducible (within 15% of the original effects or within .05 of original p-values) after inverse weighting each of the 551 claims by the number of claims per paper. We observed higher reproducibility for papers from Political Science and Economics than other fields, for more recent than older papers, and for papers from journals that required data sharing. Implementing measures to verify that research is reproducible is needed to support trustworthiness in the complex enterprise of knowledge production.

Authors

Institutions

Publication Details

Published
2026-04-01
DOI
https://doi.org/10.31222/osf.io/wqt9u_v1
Primary Topic
Meta-analysis and systematic reviews
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Investigating the reproducibility of the social and behavioural sciences

Jing Geng, Hansika Kapoor, Kamil Izydorczak, Jan Philipp Röer et al.
Meta-analysis and systematic reviews
article

Investigating the reproducibility of the social and behavioural sciences

Jing Geng, Hansika Kapoor, Kamil Izydorczak, Jan Philipp Röer, Thomas Ostermann, Nigel Mantou Lou, Ilka H. Gleibs, Marta Kołczyńska, Eli Awtrey, Kent Jason G. Cheng, Dmitry Grigoryev, Shilaan Alzahawi, Andrew H. Tyner, Tabaré Capitán, Daniel C. Molden, An-Chiao Liu, Ljiljana B. Lazarević, Mariah F. Purol, Kevin Esterling, Victor Gay, William J. Chopik, Krystal Hahn, Angelo Christopher Limnios, Tom Coupé, Olivia Miske, Macie Daley, Elif Gizem Demirag Burak, Johanna Gereke, John Wills Lloyd, Paul H. P. Hanel, Kai J. Jonas, Laura Caquelin, Alexander Olof Savi, Thomas Rhys Evans, Henk Berkman, Sara Capitán, Claudia Neuendorf, Jianhua Duan, Nicholas William Fox, Pavol Kačmár, Noah Haber, Johannes Michalak, Zoran Pavlović, Marcel Martončik, Sean Dae Houlihan, Nathaniel D. Porter, Nicholas Gunby, Kim Peters, Nathan Fiala, Brinna Mawhinney, Abiola Oyebanjo, Theresa Stankov, Marco Ramljak, Ernest O'Boyle, Kristin Jankowsky, Erick Axxe, Zachariah Berry, Austin Lee Nichols, Katelin E. Leahy, Saule Anafinova, Timothy F. Brady, David Kretschmer, Arran T Reader, Gwen-Jirō Clochard, Mirka Dirzo, Sanghyun Hong, Faisal Mushtaq, Gustav Nilsonne, Nate Breznau, Ivan Ropovik, Bert N. Bakker, Kathleen Schmidt, František Bartoš, Sebastian Karcher, W. Robert Reed, James Field, Noemi Peter, Nick Huntington-Klein, Richard Lucas, Nikita P Mehta, Melissa Kline Struhl, Radoslaw Panczak, Daniel J. Mallinson, James Bailey, Priya Silverstein, Matus Adamkovic, Anna Lou Abatayo, Amélie Gourdon-Kanhukamwe, Maximilian Maier, Jessica C. Lee, Jeewon Oh, Ruben Lopez-Nicolas, Esteban Méndez-Chacón, Gabriel Baník, Akshaya Balaji, Arathy Puthillam, Michael C. McCall, Jamie Cummins, Felix Bethke
article en

Abstract

Published claims should be reproducible, yielding the same result when applying the same analysis to the same data. We assessed reproducibility in a stratified random sample of 600 papers published from 2009 to 2018 in 62 journals spanning the social and behavioural sciences. Authors of 144 (24.0% [95% CI 20.8 - 27.6%]) papers made data available to assess reproducibility, and for 38 others, we obtained source data to reconstruct the dataset. We assessed 143 of the 182 available datasets and found that 76.6 (53.6% [95% CI 45.8 - 60.7%]) papers were rated as precisely reproducible and 105.0 (73.5% [95% CI 66.4 - 80.0%]) papers as at least approximately reproducible (within 15% of the original effects or within .05 of original p-values) after inverse weighting each of the 551 claims by the number of claims per paper. We observed higher reproducibility for papers from Political Science and Economics than other fields, for more recent than older papers, and for papers from journals that required data sharing. Implementing measures to verify that research is reproducible is needed to support trustworthiness in the complex enterprise of knowledge production.

Center for Open Science (US), Arizona State University (US), Massachusetts Institute of Technology (US), Virginia Tech (US)
Openalex Percentile: Top 39%
Meta-analysis and systematic reviews
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.