Sources of bias in general linear models: evaluating the analytic practice in psychological research

Abstract Unbiased estimation and hypothesis testing in general linear models (GLMs) require several conditions to be met, such as adherence to statistical assumptions and absence of influential data points. This study examined the extent to which researchers in psychology attend to these sources of bias when presented with analytic tasks. We recruited 188 psychology researchers who completed two data analytic scenarios. Researchers' analytic scripts and descriptions were subsequently coded based on their level of attentiveness to issues with non-normal model errors, heteroscedasticity, outliers, and influential cases. We found that for all categories, researchers either performed no checks for potential issues or performed checks that were insufficient to detect the issue at hand. Researchers who teach research methods were more likely to perform the correct checks than non-teaching faculty, post-doctoral researchers and PhD students; however, this difference was small and the highest posterior density intervals overlapped substantially. We discuss the implications of routinely neglecting violated assumptions and influential cases and present the case for more frequent application of robust statistical methods to supplement or replace ordinary least squares GLMs.