Non‐Response Insights With Binary Variables

Summary Non‐response brings into question the critical ability of probability surveys to make unbiased estimates of population parameters and to estimate the reliability of these estimates. This article considers the bias of estimates in a simple setting to help clarify some of the complex issues arising from non‐response. We show the limitation of focusing solely on the correlation between the outcome variable and the propensity to respond without considering the distribution of the response propensities. The relationship between the survey response rate and non‐response bias is also reconsidered. An important practical result is that data collection efforts to increase response rates moderately may not substantially decrease non‐response bias for most statistics. Our results support other studies showing non‐response adjusted estimates remain subject to non‐response bias and that current inference procedures that act as if the bias is eliminated produce confidence intervals that are too narrow. The findings show that proposed adjustments such as doubling margins of error may not be adequate to achieve nominal levels in some cases.