Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise ('beginners') updated their estimates and confidence in their judgements significantly more than groups with greater task expertise ('experienced'). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners' average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of 'fast science' under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.

Authors

Institutions

Publication Details

Journal
Nature Human Behaviour
Published
2024-12-20
DOI
https://doi.org/10.1038/s41562-024-01961-1
Citations
3
Primary Topic
Misinformation and Its Impacts
Type
article
Field-Weighted Citation Impact
3.40

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Hansika Kapoor, Anna Lou Abatayo, Erin Michelle Buchanan, Fiona Fidler et al.
3 citations
Nature Human Behaviour
Misinformation and Its Impacts
3.40
article

Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Hansika Kapoor, Anna Lou Abatayo, Erin Michelle Buchanan, Fiona Fidler, Marta Kołczyńska, John R. Kerr, Angela M. Jones, Kent Jason G. Cheng, Andrew H. Tyner, Tabaré Capitán, Brianna Luis, Ernest Baskin, Claudia R. Schneider, Alexandra L. J. Freeman, Melissa Kline Struhl, Raquel Huggins, Thomas Pfeiffer, Anna Fedor, John E. Edlund, Tom Coupé, Olivia Miske, Zachary Loomas, Nick Huntington‐Klein, James G. Field, Melba Tutor, Jon Roozenbeek, Ginny Chan, Felix Holzmeister, Duan Jian-hua, Anirudh Tagat, Timothy M. Errington, Zhongchen Song, Henk Berkman, Esteban Méndez, Sara Capitán, Sanghyun Hong, Fallon Mody, W. Robert Reed, Nicholas William Fox, Anca M. Hanea, Magnus Johannesson, Sander van der Linden, Andrew Soh, Ans Vercammen, David P. Wilkinson, Karolina Urbanska, Bonnie C. Wintle, Alexa R. Schlyfestone, Brian A. Nosek, Elan Simon Parsons, Alexandru Marcoci, Carolin Nast, Hannah Fraser, Sarah Dryhurst, Yang Liu
article en
3 citations

Abstract

Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise ('beginners') updated their estimates and confidence in their judgements significantly more than groups with greater task expertise ('experienced'). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners' average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of 'fast science' under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.

Nature Human BehaviourVol. 9(2)
University of Hawaiʻi at Mānoa (US), West Virginia University (US), Statistics New Zealand (NZ), Pennsylvania State University (US), Texas State University (US), University of Connecticut (US), University of Nottingham (GB), Harrisburg University of Science and Technology (US), University of Auckland (NZ), Rochester Institute of Technology (US), The University of Queensland (AU), The University of Melbourne (AU), Stockholm School of Economics (SE), University of California, Santa Cruz (US), University of Canterbury (NZ), Universität Innsbruck (AT), Curtin University (AU), University of Cambridge (GB), Center for Open Science (US), Swedish University of Agricultural Sciences (SE), Institute of Political Studies (PL), University College London (GB), Saint Joseph's University (US), Massey University (NZ), University of Virginia (US), Seattle University (US), Massachusetts Institute of Technology (US), University of Otago (NZ), Wageningen University & Research (NL), University of Stavanger (NO), Polish Academy of Sciences (PL)
U.S. Department of Defense, Defense Advanced Research Projects Agency, Advanced Research Projects Agency
Openalex Percentile: Top 7%
Misinformation and Its Impacts
3.40
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.