Investigating the replicability of the social and behavioural sciences

We attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioral sciences. Replications were high-powered on average to detect the original effect size (Median = 99.6%), used original materials when relevant and available, and were peer-reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% [95% CI 49.2 - 60.9%]) and for 80.8 of 164 papers (49.3% [95% CI 43.8 - 54.7%]) weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5% to 63.1%) though some estimates had high uncertainty. For claims with Pearson’s r effect sizes, the median was 0.25 [95% CI 0.21 - 0.27] for original studies and 0.10 [95% CI 0.09 - 0.13] for replication studies, a 82.4% [95% CI 67.8 - 88.2%] reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median = 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean due to replicating only positive results. The conditions that promote or inhibit replicability are worthy of additional investigation.

Authors

Institutions

Publication Details

Published
2026-04-01
DOI
https://doi.org/10.31222/osf.io/b7x8f_v1
Primary Topic
Meta-analysis and systematic reviews
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Investigating the replicability of the social and behavioural sciences

Nathan M. Fong, Nicholas Berente, Jing Geng, Anna Lou Abatayo et al.
Meta-analysis and systematic reviews
article

Investigating the replicability of the social and behavioural sciences

Nathan M. Fong, Nicholas Berente, Jing Geng, Anna Lou Abatayo, Christian Bokhove, Daniel J. Dunleavy, David G. Dobolyi, Ilka H. Gleibs, Eli Awtrey, Martina Grunow, Miles Bader, Kent Jason G. Cheng, Sandra J. Geiger, Robbie C. M. van Aert, Andrew H. Tyner, Tabaré Capitán, Flávio Souza Azevedo, Robin Bouwman, Lusine Grigoryan, Christine Cozette Comer, Marin Dujmović, Braeden Forrester Hall, Melissa Kline Struhl, Anna Fedor, Kevin Esterling, William J. Chopik, Krystal Hahn, Cécile Emery, Joshua R. de Leeuw, Tom Coupé, Olivia Miske, Macie Daley, Laura M. Getz, Miguel A. Fonseca, James G. Field, Gabriel Briceño Jiménez, Sam Harper, Paul H. P. Hanel, Janaki Gooty, Thomas Rhys Evans, Courtney K. Soderberg, Rahul Chawdhary, Jędrzej Białkowski, Sara Capitán, Jianhua Duan, James Druckman, Nicholas William Fox, Victoria E. Colvin, Jesse Chandler, Noah Haber, Traci M. Bricka, Roman Briker, Linda Geven, Jeremy Freese, Nicholas Gunby, Giulio Costantini, Samuel Field, Nathan Fiala, Diane B. V. Bonfiglio, Ahmed Nadeem Abbasi, Brinna Mawhinney, Cristina Greculescu, Theresa Stankov, Marjan Bakker, Chris L Aberson, Donna Pamella Gonzales, Laura Boeschoten, Tessa Charles, Alexandra L.J. Freeman, Bruce Clark, Erick Axxe, Zachariah Berry, Erin E. Hannon, Kathryn Caldwell, Sophie M. Behr, Nihan Albayrak, Aneta Czernatowicz-Kukuczka, Timothy F. Brady, Michael Andreychik, Balazs Aczel, Belén Fernández, Annette Nicole Brown, Christopher R. Chartier, George Christopher Banks, Bojana Bodroža, Peter Allen, James Bailey, Patrick Durkee, Priya Silverstein, Matus Adamkovic, Bence Bago, Amélie Gourdon-Kanhukamwe, Gabriel Baník, Ernest Baskin, Siobhan Griffin, Scott R. Braithwaite, Jamie Cummins, Gordon David Alexander Brown, Cameron Brick
article en

Abstract

We attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioral sciences. Replications were high-powered on average to detect the original effect size (Median = 99.6%), used original materials when relevant and available, and were peer-reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% [95% CI 49.2 - 60.9%]) and for 80.8 of 164 papers (49.3% [95% CI 43.8 - 54.7%]) weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5% to 63.1%) though some estimates had high uncertainty. For claims with Pearson’s r effect sizes, the median was 0.25 [95% CI 0.21 - 0.27] for original studies and 0.10 [95% CI 0.09 - 0.13] for replication studies, a 82.4% [95% CI 67.8 - 88.2%] reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median = 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean due to replicating only positive results. The conditions that promote or inhibit replicability are worthy of additional investigation.

Florida State University (US), University of Notre Dame (US), Cornell University (US), Center for Open Science (US), Virginia Tech (US)
Openalex Percentile: Top 40%
Meta-analysis and systematic reviews
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.