Comparability of YouGov and Qualtrics samples in survey experiments: A short report

Abstract Objectives There has been an increase in the use of nonprobability online samples in survey research, with studies demonstrating comparability of both YouGov and Qualtrics samples to probability-based samples, like the GSS, when it comes to multivariate estimates. Thus, we considered whether YouGov and Qualtrics are comparable to one another given the varying price points for the two platforms, as well as whether this comparability extends to treatment effects in survey experiments. Methods In a recent survey, we fielded the same vignettes centered on plea decision-making to a YouGov and Qualtrics sample. We focused on one of these vignettes to compare the two samples with respect to demographic margins, demographic intersections, treatment effects, and treatment heterogeneity. Results We found that both platforms produced consistent treatment effects, with the Qualtrics sample more comparable to YouGov’s unweighted sample than weighted sample. Comparability varied more when considering treatment heterogeneity by marginal demographic characteristics and interactive demographic characteristics, particularly with respect to political orientation. Conclusion Given the comparability found, it would be reasonable to use Qualtrics for its lower price point. That said, YouGov is the superior outlet if there is an expectation of treatment effect heterogeneity by political orientation, as is often the case in public opinion research.