Does it matter how we measure conspiracy beliefs? A test of three measurement approaches

Abstract Objectives In this article, we examine the extent to which respondents’ support for conspiratorial claims depends on question format. Measuring conspiracy theory beliefs in the general population is challenging, and properly capturing these beliefs is necessary if we are to understand them. Methods We conducted a preregistered vignette experiment of Canadian respondents (N = 3,518) in February 2024. In our experiment, we introduced survey respondents to conspiratorial claims in one of three ways: (1) by asking them whether they support a conspiratorial claim directly through a Likert scale with the response set from very likely to not likely at all, (2) by offering respondents a binary choice between a conspiratorial claim and an alternative claim, and (3) by offering respondents a trichotomous choice between a conspiratorial claim, an alternative claim, and an equally likely option. Results We find the trichotomous format produces the most conservative estimates of conspiracy endorsement, while the Likert format produces the most permissive estimates. In some cases, the percentage of respondents who endorsed conspiracy theories in the Likert questions was more than three times as high as in the trichotomous format, and in many cases was around twice as large. Conclusion Different question formats lead to substantially different estimates of conspiracy beliefs. While we believe that it is impossible to create completely bias‐free questions measuring conspiracy belief, researchers must acknowledge the likely biases (in particular, social desirability bias and acquiescence bias) that may be present in their survey designs.