A Meta-Analysis of the Impact and Heterogeneity of Explicit Demand Characteristics

Demand characteristics are a fundamental methodological concern in experimental psychology. Yet, little is known about the direction, magnitude, and consistency of their effects. We conducted a three-level meta-analysis of 253 effect sizes from 53 studies that manipulated explicit demand characteristics (EDCs). On average, EDCs led to small overall increases in hypothesis-consistent responding (g = 0.21, 95% CI [0.11, 0.30]). However, the effects were heterogeneous (between-study τ = 0.29; within-study σ = 0.18), with the prediction interval ranging from g = 0.89 (a large increase in hypothesis-consistent responding) to g = -0.48 (a moderate decrease in hypothesis-consistent responding). Consistent with previous theorizing, the observed and estimated distribution of effects suggest that demand characteristics can create false positives, false negatives, upward bias, and downward bias. These unpredictable inferential consequences suggest that further research is needed to test mechanisms theorized to underlie the effects of demand characteristics.