A Bayesian Framework to Establish Validity Evidence for Multi-Unidimensional Instruments With Small Samples

Accurate estimation of latent traits is critical in educational and psychological measurement, yet establishing validity evidence is challenging under small samples, ordinal data, and skewed distributions. We propose a Bayesian framework that incorporates expert-informed priors to enhance estimation and validity evidence for multi-unidimensional instruments. Simulation studies demonstrate that expert input improves estimation accuracy over ordinal confirmatory factor analysis, especially in samples as small as N = 25, with diminishing returns beyond six to nine experts. We compare variational inference (automatic differentiation variational inference) and Hamiltonian Monte Carlo in terms of estimation accuracy, computational efficiency, and posterior quality. A real-world application using TIMSS Grade 8 Mathematics data illustrates practical implications for expert selection and estimation strategies in small-sample instrument development.