Examining the replicability of online experiments selected by a decision market
Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015-2018, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with 2 randomly selected studies. The replication rate, based on the statistical significance indicator, was 83% for the top-12 and 33% for the bottom-12 group. Overall, 54% of the studies were successfully replicated, with replication effect size estimates averaging 45% of the original effect size estimates. The replication rate varied between 54% and 62% for alternative replication indicators. The observed replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments.
Authors
- Jürgen Huber (ORCID: https://orcid.org/0000-0003-0073-0321)
- Rene Schwaiger (ORCID: https://orcid.org/0000-0003-2607-5986)
- Vivien Margaret Walden
- Alexandra Sarafoglou (ORCID: https://orcid.org/0000-0003-0031-685X)
- Lawrence Jin (ORCID: https://orcid.org/0000-0003-3679-874X)
- Alexander Ly (ORCID: https://orcid.org/0000-0003-3925-3833)
- Gideon Nave (ORCID: https://orcid.org/0000-0001-6251-5630)
- Anna Dreber (ORCID: https://orcid.org/0000-0003-3989-9941)
- Colin F. Camerer (ORCID: https://orcid.org/0000-0003-4049-1871)
- Noboru Imai (ORCID: https://orcid.org/0009-0004-7115-5509)
- Thomas Pfeiffer (ORCID: https://orcid.org/0000-0002-0592-577X)
- Dylan Manfredi (ORCID: https://orcid.org/0000-0003-3621-9402)
- Teck‐Hua Ho (ORCID: https://orcid.org/0000-0001-5210-4977)
- Taisuke Imai (ORCID: https://orcid.org/0000-0002-0610-8093)
- Felix Holzmeister (ORCID: https://orcid.org/0000-0001-9606-0427)
- Eric‐Jan Wagenmakers (ORCID: https://orcid.org/0000-0003-1596-1034)
- Magnus Johannesson (ORCID: https://orcid.org/0000-0001-8759-6393)
- Michael Kirchler (ORCID: https://orcid.org/0000-0002-5416-2545)
- Suzanne Hoogeveen (ORCID: https://orcid.org/0000-0002-1304-8615)
- Brian A. Nosek (ORCID: https://orcid.org/0000-0001-6797-5476)
- Benjamin Mandl
- Yiling Chen (ORCID: https://orcid.org/0000-0002-0166-594X)
Institutions
- California Institute of Technology (US)
- Swedish National Bank (SE)
- Centrum Wiskunde & Informatica (NL)
- Harvard University (US)
- Osaka University of Economics (JP)
- National University of Singapore (SG)
- Nanyang Technological University (SG)
- Stockholm School of Economics (SE)
- Universität Innsbruck (AT)
- Utrecht University (NL)
- Center for Open Science (US)
- Massey University (NZ)
- University of Virginia (US)
- University of Pennsylvania (US)
- University of Amsterdam (NL)
Publication Details
- Journal
- Nature Human Behaviour
- Published
- 2024-11-19
- DOI
- https://doi.org/10.1038/s41562-024-02062-9
- Citations
- 12
- Primary Topic
- Behavioral Health and Interventions
- Type
- article
- Field-Weighted Citation Impact
- 4.92
Funders
- Jan Wallanders och Tom Hedelius Stiftelse samt Tore Browaldhs Stiftelse
- Sveriges Riksbanken
- Austrian Science Fund
- Knut och Alice Wallenbergs Stiftelse
- Riksbankens Jubileumsfond
- Marcus och Amalia Wallenbergs minnesfond