Investigating the analytical robustness of the social and behavioural sciences

The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science1–3. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts’ choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, where for one claim per study, at least five re-analysts independently re-analysed the original data. The statistical appropriateness of the re-analyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent re-analyses yielded the same result (within a tolerance region of +/- 0.05 Cohen’s d) as the original report; with a four times broader tolerance region, this indicator rose to 57%. Regarding the conclusions drawn, 74% of analyses were reported to arrive at the same conclusion as in the original investigation; 24% to no effects/inconclusive result, and 2% to the opposite effect as in the original investigation. This exploratory study suggests that the common single-path analyses in social and behavioural research should not simply be assumed to be robust to alternative analyses4. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty.

Authors

Publication Details

Journal
ORCA Online Research @Cardiff (Cardiff University)
Published
2025-10-31
Primary Topic
Meta-analysis and systematic reviews
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Investigating the analytical robustness of the social and behavioural sciences

Ali H. Al‐Hoorie, Kamil Izydorczak, Dan Mayer, Noelin Fox et al.
ORCA Online Research @Cardiff (Cardiff University)
Meta-analysis and systematic reviews
article

Investigating the analytical robustness of the social and behavioural sciences

Ali H. Al‐Hoorie, Kamil Izydorczak, Dan Mayer, Noelin Fox, Enrico Toffalini, Matthew Zawistowski, Alexandra Sarafoglou, Dmitry Sergeevich Grigoryev, Yuri G. Pavlov, Shilaan Alzahawi, Morton Ann Gernsbacher, Ellyse McCormick, Denise Maria Guimarães Freire, Jason W. Karl, Yakup Kalaycı, Alfin Leo, Timo Gnambs, Mirosława Pietryka, Aaron L. Wichman, Jeffrey A. Edlund, William E. Davis, Yoshihiko Kunisato, Markus Weinmann, Christopher L. Aberson, C. Rioux, Joshua R. de Leeuw, Robert G. M. Spencer, J.-A. Salamon, Luis Eudave, Helena Melero, Theiss Bendixen, Diogo M. Geraldes, Leonardo Tozzi, Russell A. Poldrack, Kenneth S.L. Yuen, Richard W. Briggs, Jan R. Landwehr, Gabriel Cepaluni, Guiomar Niso, Felix Holzmeister, Krisna Adiasto, Arun Sondhi, Lea Ellwardt, Gustav Nilsonne, Jorge N. Tendeiro, Sébastien Massoni, Maria Pantazi, Marie Mack, Andreas Jahn, Martina Sladekova, Ilse L. Pit, Agata Gąsiorowska, Vanina G. Franco, József Arató, Qinna Shen, Mehmet Harma, Fabrício Mendes Fialho, Noah N. N. van Dongen, Carl Michael Galang, Filip Děchtěrenko, Magdalena B. Wozniak, Darinka Trübutschek, Phillip H. McKee, Carlotta Giani, Mahmoud Medhat Elsherif, Enrico Glerean, M. S. Altman, Alyosha Boldt, Marcell Püski, Ian Wang, Harry T. Clelland, Lukas Schulze, Andrew P. Heim, Kenny van Lieshout, Livia Kosa, Ensar Acem, Barnabas Szaszi, ET AL (the CMS ECAL Group), Love Ahnström, Marton Kovacs, Hannah Schulz-Kumpel, Timofey Adamovich, Matus Adamkovic, Yousuf Abdelfatah, Zoltan A. Torma, Don Van Ravenzwaalj, Oguz A. Acar, Sabine Hoffman, L. Vaughn, A. Dorrough, B. Clarke, R. Seri, L. Steckermeier, M. Kirchler, M. Okan, M. Wall, K. Huth, S. Schwartz, M. Fišar, R. Ross, X. Lyu, V. Krivoshchekov, M. Martinoli, T. Marghetis, E. Talbert, A. Geminiani, D. Dianovics, N. Schwitter, R. Johnson, R. Zultan, E. Maassen, J. Lu, M. Mayer, M. Reshetnikov, E. Ferrouhi, I. Plank, C. Oswald, R. López-Bueno, E. Buchanan, J. Karch, A. Masiliunas, E. Ambrosini, L. Dong, A. Cataldo, A. Peez, J. Van Assche, X. Zhao, A. Mädebach, T. Schonberg, X. Schmalz, L. Graf-Vlachy, A. Plachti, W. Youyou, D. Schultz, M. Samahita, S. Pauer, C. Luo, F. Bartoš, J. Guassi Moreira, A. Sangchooli, J. Breuer, T. de Vries, C. Bosch-Rosa, H. Brohmer, S. Lo Iacono, B. González-Bustamante, J. Chandler, J. Jang, A. Farrera, C. Ziller, W. Jiménez-Leal, D. Jang, A. Eerland, K. Ong, S. Dohle, I. Rafaï, A. Dalla Rosa, J. Klein, A. Visalli, P. Vierus, M. Schaerer, C. Morvinski, N. Hajimoladarvish, R. Lopez-Nicolas, M. Lange, F. Rusche, J. Pugel, M. Grieder, J. Bastiaansen, C. Koba, K. Banas, Z. Kekecs, A. Hafenbrack, A. Furnas, M. Aviña, Y. Mora, M. Burghart, B. Kiss-Dobrony, E. Uhlmann, A. Stamos, S. Cook, V. Cafarelli, C. Guichet, D. Lempert, M. Angerer, S. Houlihan, A. Goreis, M. Ernst, D. Marino Fages, H. Lee, J. Kantorowicz, K. Miroshnik, L. Raes, E. Naumann, X. Chen, I. Jackson, C. Hughes, R. van Veldhuizen, K. Schmidt, J. Elder, B. Peterson, J. Levin, M. Kolouch Grabovský, C. Huber, A. Kazlou, J. Suchow, C. Merkle, C. Häffner, N. Byrd, S. Lee, V. Gligorić, S. Pandey, L. Lindsay, G. Mikolajczak, O. Robertson, M. Pollmann, R. Román-Caballero, M. Uhlich, H. Karimi-Rouzbahani, E. Wagenmakers, S. Liu, F. Luebber, P. Lodder, C. Czymara, J. Buckenmaier, J. Traczyk, E. Lesage, M. Kowal, I. McDonough, J. Pfänder, R. Nennstiel, I. Ropovik, A. Krypotos, A. Sahm, A. Karabulut, S. Garrison, I. Marchetti, B. Schuetze, Y. Seetahul, W. Mill, M. Desgroseilliers, J. Dora, M. Haigh, R. Heiberger, M. Kołczyńska, P. Meier, B. Scoggins, M. Varga, J. Hogeveen, V. Suschevskiy, S. Steinkamp, M. Ingendahl, R. Puri, M. Michaelides, T. Liu, Y. Wang, A. Pirrone, D. Wisniewski, J. Hicks, M. Jain, A. Onicas, S. Jungkunz, M. Máčel, P. Callahan, J. Fossum, K. Wang, Ž. Korda, S. Shaw, P. Flores-Kanter, J. Buckley, P. Szecsi, E. Balayan, K. Raiber, A. Jones, K. Carriere, A. Meier, C. Siemroth, V. Diveica, A. Trinidad, C. Baldwin, A. Rotella, L. Fiala, H. Chen, M. Neugebauer, A. Arenas, M. Scandola, J. Fiechter, L. Stasielowicz, P. Kačmár, C. Chang, E. Efendic, N. Hajdu, F. Votta, M. Folwarczny, J. Phillips, E. Baskin, L. Gurgand, F. Azevedo, M. Korbmacher, F. De Keyzer, M. Day, D. Zimmerman, B. Ganschow, C. Bauman, T. Yang, G. Navarrete, A. Somo, M. Testori, C. Westheide, A. Antonietti, E. Lemay Jr., M. Liu, J. Reynolds, A. Kažemekaitytė, S. Berger, R. Gauriot, A. Angulo-Brunet, D. Hernaus, J. Lukavský, P. Millroth, C. Moreh, S. Pawel, J. Maffly-Kipp, S. Yeung, N. Harp, J. Ludwig, M. Ploner, J. Collins, M. McIntyre, A. Pua, O. Bozkurt, F. Rüffer, S. Nebe, B. Petruželka, M. Rusconi, A. Nedderhoff, C. Cohen, L. Torkkeli, S. Grinschgl, J. Gaebler, F. Chen, Ø. Hernaes, P. Papale, A. Bouyamourn, K. Rigaud, A. Waldendorf, G. Corlazzoli, S. Chen, S. Porcher, T. Ishii, Q. Beazer, D. Lacko, O. Isler, A. Stoevenbelt, P. Allen, S. Anafinova, M. Walker, E. Nicolaisen-Sobesky, L. Brehm, M. Schreiner, R. Debnath, K. Urbanska, M. Martínez, J. Kim, A. Davis, P. Li, E. Szumowska, S. Zhou, T. Nagy, K. Turek, Y. Nielsen, R. Freichel, J. Tang, P. Bloom, E. Voldal, B. Nosek, E. Estrada, C. Kaiser, S. Mayer, R. Merz, H. Habibnia, E. Demiral, J. Höpler, H. Wallen, P. Michaelsen, C. Davis, M. Vezzoli, A. Bernard, B. Butt, I. Russo, F. Jia, F. Hagemeister, E. Schares, R. Anand, V. Chirkov, S. Rivera, H. Schaak, D. Shanks, A. Jakubow, B. Nowak, T. Heyman, H. Du, J. Lin, B. Baker, M. Weiss, M. Kubsch, J. Buczny, T. Capitán, Y. Pavlov, K. Otten, A. Kretzschmar, S. Dochow-Sondershaus, A. Tacconelli, J. Hall, N. Bault, T. Hummler, J. Frese, C. Nater, K. Mathieu, G. Pfuhl, H. Peetz, J. Petter, R. Szekely-Copîndean, A. Lovakov, J. Shaw, B. Williams, F. Venczel, T. Woiczyk, J. Wright, U. Bernardic, Á. Hartvig, S. Rathje, H. Sunde, S. Swain, M. Jekel, B. Gasparotto Ponne, K. Klasmeier, J. Wulff, R. Botvinik-Nezer, D. Grüning, A. Tomašević, C. Zogmaister, S. Topel, D. Cialfi, A. Schubert, L. Gajary, T. Evans, C. Reynolds, L. Röseler, D. Lisovoj, H. Schilling, J. Cummins, L. Spantig, B. Sonmez, B. Banai, J. Sundquist, E. Chan, S. Coelho, M. Hamjediers, Z. Zheng, A. Dreber, K. Loter, J. D'hondt, B. Bago, S. Radkani, D. Lee, M. van Assen, T. Alsalti, M. Misiak, A. McGowan, R. Zein, M. Bachl, C. Street, D. Moreau, M. Marini, A. Godefroidt, P. Kolias, A. Sanaei, J. Schröder
article en

Abstract

The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science1–3. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts’ choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, where for one claim per study, at least five re-analysts independently re-analysed the original data. The statistical appropriateness of the re-analyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent re-analyses yielded the same result (within a tolerance region of +/- 0.05 Cohen’s d) as the original report; with a four times broader tolerance region, this indicator rose to 57%. Regarding the conclusions drawn, 74% of analyses were reported to arrive at the same conclusion as in the original investigation; 24% to no effects/inconclusive result, and 2% to the opposite effect as in the original investigation. This exploratory study suggests that the common single-path analyses in social and behavioural research should not simply be assumed to be robust to alternative analyses4. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty.

ORCA Online Research @Cardiff (Cardiff University)
Reduced inequalities
Openalex Percentile: Top 25%
Meta-analysis and systematic reviews
OpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.