The reliability of replications: a study in computational reproductions

This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and <i>p</i>-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.

Authors

Institutions

Publication Details

Journal
Royal Society Open Science
Published
2025-03-01
DOI
https://doi.org/10.1098/rsos.241038
Citations
5
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
article
Field-Weighted Citation Impact
12.40

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

The reliability of replications: a study in computational reproductions

Salman Khan, Arno Van Hootegem, Jonathan Mijs, Theresa Gessler et al.
5 citations
Royal Society Open Science
Explainable Artificial Intelligence (XAI)
12.40
article

The reliability of replications: a study in computational reproductions

Salman Khan, Arno Van Hootegem, Jonathan Mijs, Theresa Gessler, Jolanda van der Noll, Juan Carlos Castillo, Christian S. Czymara, Daniel Nüst, Alexander Wuttke, Anne-Kathrin Stroppe, Dafina Kurti Sinatra, Sebastian Jungkunz, Brenton M. Wiernik, Yuki Yamada, Erich Striessnig, Nadia Steiber, Hannah Werner, Laura Jacobs, Nate Breznau, Magdalena Hirsch, Marta Kołczyńska, Kathrin Busch, Klaus Burger, Hung Hoang Viet Nguyen, Jasper Van Assche, Simon Ellerbrock, Jan Paul Heisig, Manuel Kleinert, Arne Roets, Merlin Schaeffer, Daniel Auer, Max Grömping, Jonathan D. Rogers, Nora Storz, Katharina Kunißen, Bogdan Voicu, Sebastian Sternberg, Philipp Lutscher, Alejandro Ecker, Johannes N. Blumenberg, Fabian Ochsenfeld, Eline A. de Rooij, Janna Teltemann, Vernon Gayle, Andrey Tibajev, Pablo Christmann, Alexander Staudt, Dragana Stojmenovska, Licia Bobzien, Thijs Bol, Jürgen Schneider, Timo Gnambs, Cecil Meeusen, Paul Cornelius Bauer, Claudia Schmiedeberg, Daniel Meierrieks, Danilo Freire, Christopher Prosser, Elmar Schlueter, Jonathan Mellon, Verena Halbherr, Stefan Vogtenhuber, Patricia McManus, Leticia Micheli, Regine Schmidt, Stefan Gruber, Willem W. A. Sleegers, Julia Klinger, Esra Akdeniz, Daniel Meyer, Sharon Baute, Ariela Schachter, Henning Silber, Dieter Stiers, Sandy Schumann, Nathan Chan, Nico Sonntag, Joan E. Madia, Oke Bahnsen, Robin Samuel, Roxanne Connelly, Meta van der Linden, Samuel Merk, Christina Eder, Zsófia S. Ignácz, Andrea Bohman, Mikael Hjerm, Verena Benoit, Jan H. Höffler, Fabian Winter, Nora Huth-Stöckle, Andreas Hövermann, Stefanie Heyne, Anna Pechenkina, Katharina Blinzler, Oshrat Hochman, David Schieferdecker, Philipp M. Lersch, Jule Adriaans, Helge Marahrens, Stefan Zins, John Kuk, Natalia Malancu, Andreas Hadjar, Christiaan Reinier Gaasendam, Mark Pickup, Dave Balzer, Robert F. McDonnell, Lea-Maria Löbel, Katharina Burgdorf, Giacomo Vagni, Sayak Khatua, Nils D. Steiner, Fieke M. A. Wagemans, Cristóbal Moya, Jennifer Kanjana, Thomas Biegert, Christian Hunkler, Felix S. Bethke, Marcel Neunhoeffer, Martijn Schoonvelde, Amélie Godefroidt, Tomasz Żółtak, Anna Forke, Frank Reichert, Mathias Kauff, Nadja Wehl, Julian Bernauer, Bastian Jaeger, Paul Martinez, Cary Wu, Björn Zakula, Brian Tung, Jannes Jacobsen, Kevin Ralston, Jordan W. Suchow, Zuzanna Brzozowska, Amie Bostic, Konstantin Gavras, Muna Adem, Luis Maldonado, Gunnar Otte, Sophia Hunger, Andrea Förster, Guido Ropers, Julia Schulte-Cloos, Constanza Sanhueza Petrarca, Henrik Kenneth Andersen, Julian Seuring, Olav Nygård, Friedolin Merhout, Oscar J. Mayorga, Tobias Gummer, Gregor Sand, Sebastian Hellmeier, Eike Mark Rinke, Alexander Langenkamp, Jan-Philipp Kolb, Nicole Martin, Elena Damian, Carl Berning, Matthias Mader, Sabine Israel, Flávio Azevedo, Alexander Schmidt‐Catran, Conrad Ziller, Nils Jungmann, Louis Raes, Maureen A. Eger, Anna Berthold, Reinhard Schunck, Jochen Mayerl, Christof Wolf, Robin C. Lee, Kyle McWagner, Amalia Álvarez-Benjumea, Nan Zhang, David Liu, Ling Bai, Katja Schmidt, Miguel R. Ramos, Gerrit Bauer, Markus Baumann
article en
5 citations

Abstract

This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and <i>p</i>-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.

Royal Society Open ScienceVol. 12(3)
Linköping University (SE), Stevens Institute of Technology (US), Boston University (US), Goethe University Frankfurt (DE), Griffith University (AU), Australian National University (AU), FernUniversität in Hagen (DE), Leiden University (NL), Utah State University (US), Uppsala University (SE), University of Copenhagen (DK), Lucian Blaga University of Sibiu (RO), German Institute for Economic Research (DE), Statistisches Bundesamt (DE), University of Vienna (AT), University of Leeds (GB), Oregon State University (US), Université Libre de Bruxelles (BE), Collegio Carlo Alberto (IT), Consejo Superior de Investigaciones Científicas (ES), University of Bonn (DE), Kyushu University (JP), Austrian Academy of Sciences (AT), Western Governors University (US), University of Antwerp (BE), Max Planck Society (DE), Emory University (US), Peace Research Institute Frankfurt (DE), University of Fribourg (CH), University of Hildesheim (DE), Witten/Herdecke University (DE), University of Freiburg (DE), Philipps University of Marburg (DE), University of California, Los Angeles (US), Pontificia Universidad Católica de Chile (CL), University of Wuppertal (DE), University of Groningen (NL), University of Potsdam (DE), University of Mannheim (DE), Simon Fraser University (CA), University of Bremen (DE), Georgetown University (US), University of Oslo (NO), University of Luxembourg (LU), University of Konstanz (DE), York University (CA), Utrecht University (NL), Tilburg University (NL), Johannes Gutenberg University Mainz (DE), Justus-Liebig-Universität Gießen (DE), Princeton University (US), Bielefeld University (DE), University of Zurich (CH), Washington University in St. Louis (US), Institut d'Etudes Politiques de Paris (FR), University of Applied Sciences Leiden (NL), Heidelberg University (DE), European University Viadrina (DE), Institut für Arbeitsmarkt und Berufsforschung (DE), Chemnitz University of Technology (DE), North-West University (ZA), University of Michigan (US), Vienna Institute of Demography (AT), Flemish Government (BE), Freedom House (US), WZB Berlin Social Science Center (DE), The University of Texas Rio Grande Valley (US), University of Manchester (GB), Munich School of Philosophy (DE), LMU Klinikum (DE), Universidad Espíritu Santo (EC), Ghent University (BE), Loyola Marymount University (US), Humboldt-Universität zu Berlin (DE), University of Illinois Chicago (US), Institute for Advanced Study (US), University of Oxford (GB), University of Chicago (US), MSH Medical School Hamburg – University of Applied Sciences and Medical University (DE), Danish National Centre for Social Research (DK), Institute for Urban and Regional Research (AT), Federal Employment Agency (DE), Swiss Federal Archives (CH), GESIS - Leibniz Institute for the Social Sciences (DE), Healthpoint (AE), Institute of Philosophy and Sociology (PL), Indiana University Bloomington (US), Karlsruhe University of Education (DE), Hans Böckler Foundation (DE), Gesundheit Österreich (AT), The Expert Council of German Foundations on Integration and Migration (DE), Centro de Ciencias Humanas y Sociales (ES), Institute of Economic and Social Development (AR), German Institute for Adult Education – Leibniz Centre for Lifelong Learning (DE), Max Planck Institute for Behavioral Economics (DE), DIPF | Leibniz Institute for Research and Information in Education (DE), Institute for Migration Research (HR), Sciensano (Belgium) (BE), Leibniz Institute for Educational Trajectories (DE), Netherlands Institute for Social Research (NL), Institut für Höhere Studien - Institute for Advanced Studies (IHS) (AT), Institute of Political Studies (PL), University of Wisconsin–Milwaukee (US), University College London (GB), Mannheim Centre for European Social Research (DE), University of Neuchâtel (CH), Romanian Academy (RO), University of Duisburg-Essen (DE), University of Maryland, College Park (US), University of Chile (CL), Marmara University (TR), Freie Universität Berlin (DE), Technische Universität Dresden (DE), University of Birmingham (GB), University of Tübingen (DE), Ludwig-Maximilians-Universität München (DE), Michigan State University (US), University of Amsterdam (NL), Umeå University (SE), London School of Economics and Political Science (GB), University of Bamberg (DE), University of Edinburgh (GB), KU Leuven (BE), Polish Academy of Sciences (PL)
Agencia Nacional de Investigación y Desarrollo
Peace, Justice and strong institutions, Reduced inequalities
Openalex Percentile: Top 2%
Explainable Artificial Intelligence (XAI)
12.40
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.