Quality Controls Can Improve the Replicability of Survey Experiments Using Online Nonprobability Panels
Scholars across multiple disciplines have documented challenges related to replicability, often referred to as a “replication crisis.” While a range of explanations exist, some have argued that the growing reliance on online nonprobability samples contributes to replication failures. At the same time, concerns have mounted regarding the quality of these samples, as inattentive, duplicate, or fraudulent respondents can bias estimates and diminish effect sizes. In this research note, we replicate three well-known survey experiments, using data collected from three separate online nonprobability panels (N = 6,000). We compare the “off-the-shelf” samples with subsets that pass a series of attention and fraud checks and that remove duplicate respondents. We find that all three experiments replicate in terms of direction and significance across the full samples, but in two out of three cases, effect magnitudes are replicated only once low-quality responses are removed.
Authors
- James Martherus (ORCID: https://orcid.org/0000-0002-8285-3300)
- Alexander R. Podkul
- Edgar Cook (ORCID: https://orcid.org/0000-0002-3045-1270)
- Anja Kilibarda (ORCID: https://orcid.org/0000-0001-8952-6110)
Publication Details
- Journal
- Public Opinion Quarterly
- Published
- 2026-11-21
- DOI
- https://doi.org/10.1093/poq/nfag073
- Primary Topic
- Survey Methodology and Nonresponse
- Type
- article
- Field-Weighted Citation Impact
- 0.00