Experiment-based calibration in psychology: Foundational and data-generating model

Experiment-based calibration is a novel method for measurement validation, which – unlike classical validity metrics – does not require stable between-person variance. In this approach, the latent variable to be measured is manipulated by an experiment, and its predicted scores – termed standard scores – are compared against the measured scores. Previous work has shown that under plausible boundary conditions, the correlation between standard and measured scores – termed retrodictive validity – is informative about measurement accuracy, i.e. combined trueness and precision. Here, I expand these findings in several directions. First, I formalise the approach in a probability-theoretic framework with the concept of a standardised calibration space. Second, I relate this framework to classical validity theory and show that the boundary conditions in fact apply to any form of criterion validity, including classical convergent validity. Thus, I state precise and empirically quantifiable boundary conditions under which criterion validity metrics are informative on validity. Third, I relate these boundary conditions to confounding variables, i.e. correlated latent variables. I show that in the limit, calibration will converge on the latent variable that is most closely related to the standard. Finally, I provide a framework for modelling the data-generating process with Markov kernels, and identify sufficient conditions under which the data generation model results in a calibration space. In sum, this article provides a formal probability-theoretic framework for experiment-based calibration and facilitates modelling and empirical assessment of the data generating processes. • A formal model for experiment-based calibration is simplified and generalised. • This reveals novel boundary conditions for classical convergent validity. • Formalising the role of confounds reveals convergence on theory-defined latent variable. • A new formal data-generating model aids future simulation and empirical work.