Assessing effect sizes, variability, and power in the web-based study of language production

With the pandemic, experimental psychologists/linguists started to collect data over the internet (“on-line”). The feasibility of such experiments and the sample sizes required to achieve sufficient statistical power must be assessed. This requires information on effect sizes and variability. We compare response times in the same word production experiment conducted in the lab and on-line to determine whether the two settings differ in effect sizes, in the consistency of responses within participants, in the variability of average response times and effect sizes across participants, or in unexplained variability. We assess the impact of these differences on the power of the design. Analyses show that effect sizes can differ substantially between settings. Response times are more variable on-line. This is partly due to larger differences in mean response times across speakers and to a higher within-participant variability. These findings suggest that on-line production studies are feasible but at a non-negligible cost.