Zooming Out On Education: Making Valid Psychological Inferences From Large-Scale Assessment Data

Abstract Openly available datasets from large-scale educational assessments like PISA, PIRLS, TIMSS, or NAEP, among others, are some of the most valuable public resources in the education sciences. Understandably, educational psychologists are interested in analyzing these datasets to advance their research. But, in contrast to the kinds of psychoeducational assessments about which educational psychologists are typically trained, large-scale assessments are not designed to make inferences about the mental attributes of students themselves, but about the population distributions of those attributes. This seemingly subtle distinction leads to a host of analytic, epistemological, and interpretative challenges that can cause confusion and dissuade educational psychologists from using these interesting datasets. In this theoretical paper, we seek to clarify the kinds of inferences that can be validly made with large-scale assessment data, and justify those inferences based on the psychometric and score-generation procedures that underpin them. This paper is not intended to be a technical or methodological guide to analyzing large-scale assessment data but instead serves as an epistemic and conceptual introduction to the topic. After appropriately accounting for various sources of error in large-scale assessment proficiency estimates, researchers can make interesting inferences about education and psychology, but those inferences can only be validly made at the population level, not about individual students.