Robust Emotion Manipulation for Surveys: Evidence from Three Experiments

Many experiments purport to identify the effects of emotions on outcomes like voting, support for authoritarians, and polarization. These experiments universally rely upon emotion manipulation instruments. The instruments are intended to manipulate target emotions in such a way that researchers can attribute effects of the instrument to effects of the emotion. However, we know little about which instruments are strong enough to justify this attribution. We know even less about what other emotions or responses are prompted by standard instruments. This paper asks which standard emotion manipulation instruments work as intended. First, using causal graphs, we highlight that identifying the effect of emotions on downstream political outcomes requires that instruments minimize violations of an implicit exclusion restriction. Second, we empirically test whether standard instruments strongly manipulate target emotions and satisfy the exclusion restriction by minimally manipulating non-target emotions. Across three experiments (total N=6,649), we study vignettes, autobiographical emotional memory tasks (AEMTs), images, and more as instruments for anger, gratitude, fear, and various political emotions. We find that vignettes generally maximize instrument strength and minimize violations of the exclusion restriction---in contrast to the popularity of AEMTs in applied research. We also find that positive pre-treatment attitudes toward research may moderate instrument effectiveness.