Sources of bias in general linear models: assessing researchers’ understanding and self-reported practice

Abstract General linear models (GLMs) are commonly employed in psychology research. GLMs require several conditions to be met to provide unbiased and optimal parameter estimation and hypothesis tests. These conditions include adherence to statistical assumptions and the absence of extreme cases. The extent to which psychology researchers attend to these sources of bias remains under-researched. Here, we present the results of a self-report survey completed by 794 self-selected psychology researchers. Participants were presented with analytic scenarios and indicated the frequency with which they attend to GLM conditions. We also surveyed researchers’ theoretical understanding of GLM conditions and the methods they use to detect and address problems. We fitted Bayesian ordinal models and used highest posterior density (HPD) intervals to compare researchers’ self-reported knowledge and statistical practice across scenarios, subject areas and involvement in teaching research methods and statistics. We found that researchers often do not attend to violations of GLM conditions, and there are gaps in their understanding of the consequences of these violations. We found no substantial differences across psychology subject areas, and the scores of research methods instructors were comparable to those of non-teaching faculty. We discuss the implications of these findings in the context of broader credibility discussions.