Trade-offs between internal and external validity in economic experiments to inform agricultural policy

Abstract Economic experiments are increasingly used to evaluate agri-environmental policies ex-ante. They typically rely on one of three subject pools: standard subject pool students, agricultural students, or farmers. While farmers are often considered most informative for policymaking, experiments outside controlled lab environments face challenges in identifying causal effects. Thus, trade-offs between internal and external validity arise but remain underexplored empirically. Using exploratory analyses, we compare conservation behaviour under a novel agri-environmental policy, four behavioural factors, and methodological measures across farmers, agricultural students, and standard subject pool students in lab, classroom, and online settings in Germany. Conservation behaviour and environmental preferences differ significantly across pools, indicating that student samples – even with agricultural backgrounds – cannot fully substitute for farmers. Yet agricultural samples also show threats to internal validity, including more extreme response times, miscomprehension, and attrition. Trade-offs are thus evident, demonstrating that experiments with standard subject pools and agricultural samples are complements rather than substitutes. We formulate design recommendations for researchers to address threats to internal validity in experiments involving agricultural samples.