Investigating the analytical robustness of the social and behavioural sciences

The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science1–3. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts’ choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, where for one claim per study, at least five re-analysts independently re-analysed the original data. The statistical appropriateness of the re-analyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent re-analyses yielded the same result (within a tolerance region of +/- 0.05 Cohen’s d) as the original report; with a four times broader tolerance region, this indicator rose to 57%. Regarding the conclusions drawn, 74% of analyses were reported to arrive at the same conclusion as in the original investigation; 24% to no effects/inconclusive result, and 2% to the opposite effect as in the original investigation. This exploratory study suggests that the common single-path analyses in social and behavioural research should not simply be assumed to be robust to alternative analyses4. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty.

Authors

Institutions

Publication Details

Published
2026-03-31
DOI
https://doi.org/10.31222/osf.io/twqsv_v1
Primary Topic
Meta-analysis and systematic reviews
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Investigating the analytical robustness of the social and behavioural sciences

Valerii Chirkov, Laurel Brehm, Daniela Cialfi, Christopher W. Bauman et al.
Meta-analysis and systematic reviews
article

Investigating the analytical robustness of the social and behavioural sciences

Valerii Chirkov, Laurel Brehm, Daniela Cialfi, Christopher W. Bauman, Annika Boldt, Anna Dalla Rosa, Marko Bachl, Shilaan Alzahawi, Johannes Buckenmaier, Alberto Antonietti, Adam Zaid Austin Bouyamourn, Tabaré Capitán, Gaia Corlazzoli, Andreu Arenas, Matthias Burghart, Jojanneke A. Bastiaansen, Ariadna Angulo-Brunet, Theiss Bendixen, Nadège Bault, Gabriel Cepaluni, Felix Holzmeister, Krisna Adiasto, Nick Byrd, Timofey Adamovich, Bradley J. Baker, Jesse Chandler, Ettore Ambrosini, Ciril Bosch-Rosa, József Arató, Taym Alsalti, Quintin Beazer, Patrick Callahan, Bilal Hayat Butt, Andrea Michael Cataldo, Valentina Cafarelli, Harry T. Clelland, Marco M. Aviña, Livia Kosa, Ensar Acem, Yousuf Abdelfatah, Zoltan A. Torma, Oguz A. Acar, Micah Altman, Chris L Aberson, Sophie G. Coelho, Ursa Bernardic, Clara Cohen, Adil S. Al-Busaidi, Saule Anafinova, Atakan M. Akil, Jeffrey Buckley, Chia-chen Chang, Fruzsina Venczel, Ali H. Al-Hoorie, Casper Albers, Rotem Botvinik-Nezer, Cassandra Baldwin, Kasia Banas, Martin Angerer, Balazs Aczel, Zoé Bernard Anna Juliette, Eugene Y. Chan, Fadong Chen, Rahul Anand, Barnabas Szaszi, Gustav Nilsonne, Love Ahnström, František Bartoš, Hao Chen, Kevin R Carriere, Sebastian Berger, Štěpán Bahník, Don van Ravenzwaaij, Johannes Breuer, Xi Chen, Marton Kovacs, Beth Clarke, Abi M. B. Davis, Sabine Hoffmann, Peter Allen, Elza Balayan, Paul Bloom, Flavio Azevedo, Hannah Schulz-Kümpel, Matus Adamkovic, Jonathan D'hondt, Bence Bago, Hilmar Brohmer, Erin Michelle Buchanan, Christian S. Czymara, Ryan C Briggs, Shirley (Shuo) Chen, Maciej Behnke, Ernest Baskin, Özge Bozkurt, Jacek Buczny, Jamie Cummins, Benjamin Banai
article en

Abstract

The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science1–3. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts’ choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, where for one claim per study, at least five re-analysts independently re-analysed the original data. The statistical appropriateness of the re-analyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent re-analyses yielded the same result (within a tolerance region of +/- 0.05 Cohen’s d) as the original report; with a four times broader tolerance region, this indicator rose to 57%. Regarding the conclusions drawn, 74% of analyses were reported to arrive at the same conclusion as in the original investigation; 24% to no effects/inconclusive result, and 2% to the opposite effect as in the original investigation. This exploratory study suggests that the common single-path analyses in social and behavioural research should not simply be assumed to be robust to alternative analyses4. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty.

Princeton University (US), Washington University in St. Louis (US), Temple College (US), Massachusetts Institute of Technology (US), Temple University (US)
Reduced inequalities
Openalex Percentile: Top 45%
Meta-analysis and systematic reviews
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.