Data quality disparities in large-scale assessments: insufficient effort responding across student groups, schools, and cultures

Abstract Although self-report surveys are widely used for data collection, data quality can vary across populations because certain groups are more likely to engage in insufficient effort responding (IER). Our study examined how different levels of the educational system—student groups, schools, and cultural contexts—affect data quality due to IER, using a three-level Poisson regression analysis of the PISA 2018 dataset. We observed IER prevalence ranging from 14 to 74% in PISA questionnaire, depending on the indicators used. Notably, students with low academic performance were more likely to engage in IER. Additionally, students from low academic performance or high-SES schools exhibited higher IER tendencies, while IER showed minimal variation across cultural contexts. These findings emphasized that IER introduced systematic biases into survey data, undermining the fairness of group comparisons and confounding results with participants’ characteristics. Our study advocated for a careful approach to addressing IER to enhance the validity of self-report measurements.