Toward a Margin of Total Error

Abstract The margin of error that is typically reported with a poll is based on sampling variability. This measure of uncertainty excludes bias from nonresponse and measurement error, which can in practice be much larger than the sampling error. We examine empirical models for predicting a “margin of total error” that includes both sampling and nonsampling error. Models considered include a multiplicative adjustment to the standard error, a linear regression model predicting squared bias from covariates, and linear and logistic mixed models that predict systematic bias components through fixed effects and incorporate extra non-sampling variability through a vector of random effects. These models are fit to two sets of polls that have benchmark estimates available from election results and high-quality probability surveys. The linear regression model and linear mixed model had the best performance for the datasets considered. These models are easy to interpret, offer flexibility for modeling the relationship between bias and the benchmark proportion, and can include covariates about sampling protocols and question topics. The oft-cited recommendation to double the margin of sampling error achieved approximately correct coverage for confidence intervals in the election dataset but achieved poor coverage for other polls. Recommendations for inflating standard errors of election polls do not necessarily carry over to polls on other subjects. The models in this article could be used to derive standard error adjustments for a database of polls having benchmark estimates. Poll producers could then draw on predictions of excess variability that have been calculated for similar poll designs and topics to provide measures of uncertainty for future polling estimates.