Catch Questions Can Increase Lookups on Political Knowledge Questions

Abstract Surveys commonly use “catch” or “trap” questions to identify subjects who are likely to look up the answers to knowledge questions. This note shows that catch questions can backfire, causing an increase in the proportion of respondents who look up the answers to subsequent questions. In two preregistered survey experiments in online convenience samples, a catch question was randomly assigned to appear before or after a knowledge battery. When the catch question appeared first, window switching on the knowledge questions was about 25 percent more common. This suggests that when they are used, catch questions should be placed at the end of knowledge batteries, and should also not precede later items that could be affected by lookups. More generally, the results highlight the importance of pre-testing catch and trap questions for unintended consequences.