Investigating the replicability of the social and behavioral sciences

We attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioral sciences. Replications were high-powered on average to detect the original effect size (Median = 99.6%), used original materials when relevant and available, and were peer-reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% [95% CI 49.2 - 60.9%]) and for 80.8 of 164 papers (49.3% [95% CI 43.8 - 54.7%]) weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5% to 63.1%) though some estimates had high uncertainty. For claims with Pearson’s r effect sizes, the median was 0.25 [95% CI 0.21 - 0.27] for original studies and 0.10 [95% CI 0.09 - 0.13] for replication studies, a 82.4% [95% CI 67.8 - 88.2%] reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median = 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean due to replicating only positive results. The conditions that promote or inhibit replicability are worthy of additional investigation.

Authors

Publication Details

Journal
London School of Economics and Political Science Research Online (London School of Economics and Political Science)
Published
2025-12-19
Primary Topic
Meta-analysis and systematic reviews
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Investigating the replicability of the social and behavioral sciences

Nathan M. Fong, Hansika Kapoor, Samuel Shaki, Kamil Izydorczak et al.
London School of Economics and Political Science Research Online (London School of Economics and Political Science)
Meta-analysis and systematic reviews
article

Investigating the replicability of the social and behavioral sciences

Nathan M. Fong, Hansika Kapoor, Samuel Shaki, Kamil Izydorczak, Susan Griffin, Christian Bokhove, Daniel Jolles, Thomas Ostermann, Sebastian Schuetz, Daniel J. Dunleavy, David G. Dobolyi, Alan Brown, Marta Kołczyńska, M. Margarette Sanchez, Eli Awtrey, Martina Grunow, Miles Bader, Landon Schnabel, Michalis P. Michaelides, Anna M. Fedor, Jeewon Oh, Sean Patrick Roche, Sandra J. Geiger, Robbie C. M. van Aert, Finnian Wort, Richard M. Shafranek, Andrew H. Tyner, Dermot Lynott, Tabaré Capitán, David F Santos, Daniel C. Molden, Miklos Bognar, Robin Bouwman, An-Chiao Liu, Mariah F. Purol, Lusine Grigoryan, Sofia Persson, Marin Dujmović, Constant Pieters, Braeden Forrester Hall, Barnabás Szászi, Susanne Täuber, Biljana Jokić, Aaron L. Wichman, Leticia Micheli, Kevin Esterling, Ignazio Ziano, David Moreau, Gail McMillan, Tom Hostler, Christopher L. Aberson, Katie A. Munt, Tom Coupé, Joshua J. Matacotta, Arijit Nandy, Jamie Patrianakos, Jason M. Prenoveau, Ernest H. O’Boyle, Lyonel Laulié, Scott A. Baldwin, James G. Field, Marian Sauter, J. L. Williams, Aneta Czernatowicz‐Kukuczka, Daniele Marinazzo, Tina Sundelin, John Wills Lloyd, Sam Harper, Paul H. P. Hanel, D. Tsaklakidou, Adoril Oshana, Eirik Strømland, Liad Mudrik, Amanda Sargent, Ignacio Pavez, Natalia Stanulewicz, Cristina Zogmaister, Marco J. Held, Anirudh Tagat, Shishir Shakya, Cristóbal Moya, Esteban Méndez, Courtney K. Soderberg, Rahul Chawdhary, Sara Capitán, Maksim Sitnikov, Bence Bagó, Laetitia B. Mulder, Jianhua Duan, James Druckman, Jonathan M. P. Wilbiks, Michelangelo Landgrave, Florian Neubauer, Rolf Antonius Zwaan, Elisabeth Julie Vargo, Anatolia Batruch, Mitchell M. Metzger, Lars Tummers, Diem Thi Hong Vo, Jesse Chandler, Matthew R. Sisco, Victor R. Volkman, Louis Hickman, Lukasz Walasek, Jason Samaha, Melba V. Tutor, Lara Warmelink, Johannes Michalak, Traci M. Bricka, Svenja Hippel, Linda Geven, Ellie Shackleton, Jeremy Freese, Annika Beatteay, Joel M. Le Forestier, Carolin Nast, Nicholas Gunby, Kim Peters, Giulio Costantini, Zorana Zupan, Marie Isabelle Weißflog, Anđela Keljanović, Rachel P. Ostrowski, Moin Syed, Sanghyun Hong, Kimberly A. Quinn, Kathryn Nason, Elise Turkovich, Leigh Ann Vaughn, Gabriel Baník, Jesper Wulff, Soorya Sankaran, Nathan Fiala, Chris Street, Ivan Ropovik, Ahmed Nadeem Abbasi, Cristina Greculescu, Margaret Ritzau, Anna Szabelska, Abiola Oyebanjo, Marjan Bakker, Marco Perugini, Danka Purić, Nicolas Sommet, Timothy M Errington, Brian A Nosek, ..., Michael Schulte-Mecklenbeck, Kent Jason Cheng, Janaki Gooty, Ewa Szumowska, Louis Tay, Sangsuk Yoon, Jamie Cummins, Bojana Bodroža, Gideon Nave, Deliah Wagner, Patrick K. Durkee, Sven Hoeppner, Yuri G. Pavlov, Roman Briker, Cillian McHugh, Donna Pamella Gonzales, Marco Ramljak, Corinna S. Martarelli, Nathaniel D. Porter, Hani Safadi, Jordan MacDonald, Eric L. Sevigny, C. Cozette Comer, Radoslaw Panczak, Ryan Wright, Dimitra Kouroupaki, Matthew Jensen, Miguel A. Fonseca, Kathleen Schmidt, Matúš Adamkovič, W. Robert Reed, Liuqig Wei, Karolina Urbanska, Michael Inzlicht, Marcel van Assen, Gabriel Briceño Jiménez, Phuong Linh L. Nguyen, Ernest Baskin, Jacob Rothschild, Justine Saal, Amber N. Schroeder, Laura Boeschoten, Jędrzej Białkowski, Fatih Sonmez, Arathy Puthillam, Michael Andreychi, Tessa Charles, Simine Vazire, Amélie Gourdon-Kanhukamwe, Cameron Brick, Ambar G. Monjaras, James Bailey, Michaela Ritchie, Laura Smalarz, Christopher Limnios, Kristin Jankowsky, Gordon D.A. Brown, Xindong Xue, Balazs Aczel, L. Robert Slevc, Samjhana Koirala, Sungmok Lee, Anna Elisabeth van 't Veer, Heinrich H. Nax, Iris Žeželj, Laura M. Getz, Marie Juanchich, Joel S. Snyder, Priya Silverstein, Yinxian Zhang, Michael J. Larson, Eric-Jan Wagenmakers, Alexandra L.J. Freeman, Savannah C. Lewis, Cécile Emery, Ilka H. Gleibs, Miroslav Sirota, Peter Allen, Belén Fernández-Castilla, Victoria E. Colvin, Elliot A. Ludvig, Jule Oettinghaus, Yuzhi Yang, Jared Leslie, Andrew McBride, Astrid Schütz, Romina Rodela, Bruce Clark, Niels van de Ven, Erick Axxe, Frank Walter, Zachariah Berry, Nihan Aydemir, Gustav Nilsonne, Hause Lin, Angela M. Jones, Erin E. Hannon, Stuti Thapa, George C. Banks, Phillip Jolly, Austin Lee Nichols, Theresa Stankov, Veronica X. Yan, Kathryn Caldwell, Bastian Jaeger, Audrey Morrow, Diane Bonfiglio, Timothy F. Brady, Nathan C. Higgins, Jason Thatcher, Jentien M. Vermeulen, Christopher R. Chartier, Sophie M. Behr, Colin Tucker Smith, Nicholas Weller, Peter Mallik, Marina Milyavskaya, Katelin E. Leahy, Nicholas Berente, Jeremy K. Millar, Alice C. E. Lawrence, Flavio Azevedo, George Stock, Ulrich Kühnen
article en

Abstract

We attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioral sciences. Replications were high-powered on average to detect the original effect size (Median = 99.6%), used original materials when relevant and available, and were peer-reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% [95% CI 49.2 - 60.9%]) and for 80.8 of 164 papers (49.3% [95% CI 43.8 - 54.7%]) weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5% to 63.1%) though some estimates had high uncertainty. For claims with Pearson’s r effect sizes, the median was 0.25 [95% CI 0.21 - 0.27] for original studies and 0.10 [95% CI 0.09 - 0.13] for replication studies, a 82.4% [95% CI 67.8 - 88.2%] reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median = 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean due to replicating only positive results. The conditions that promote or inhibit replicability are worthy of additional investigation.

London School of Economics and Political Science Research Online (London School of Economics and Political Science)
Openalex Percentile: Top 28%
Meta-analysis and systematic reviews
OpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.