Comparative quality indicators for capturing latent constructs with single questions

Slides from the Talk: Singh, R. K. (2025). Comparative quality indicators for capturing latent constructs with single questions.European Survey Research Association (ESRA) 2025 Conference, Utrecht University, Utrecht, 2025-07-15. https://www.europeansurveyresearch.org/conf2025/prog.php?sess=184#76 Abstract: It is challenging to assess the quality of single questions for latent constructs, such as values, opinions, or interests. Psychometry instead uses multiple indicators, which allow us to judge measurement quality with factor analyses and measures of internal consistency. However, for single questions we cannot rely on the same methods.I will argue that we can sidestep some of these limitations by comparing different single questions for the same construct. I transfer a construct validation approach proposed by Westen and Rosenthal to the realm of single-item measures in social science surveys. Building upon research on ex-post harmonization, I show that we can learn much about either question by comparing how they both correlate to the set of covariates. The method gives us a metric of how similar the constructs are that both questions measure and a coefficient that quantifies the relative reliabilities of both questions. Note that we do not need answers to both questions from the same set of respondents. We merely need two independent random samples from the same population; one for each question. A structure akin to split-ballot experiments.The talk extends research presented at the last ESRA conference by exploring the two metrics with a set of simulations, a larger survey data set, and by applying it to data quality assessment (instead of data integration). Aside from demonstrating the validity of both metrics, I aim to show pragmatic use cases. For example, demonstrating empirically that we can use quite different questions (e.g., interest in political TV shows) as serviceable proxies for some constructs (e.g., general political interest). I also show that if we have quantified the reliability of one question, we can leverage that information to predict the quality of another question on the same construct.