Validation as evaluating causal validity arguments using CFA : demonstration with PIRLS reading assessment

Validity is the most fundamental consideration in developing and evaluating tests. Confirmatory factor analysis (CFA) is widely used to obtain evidence based on the internal structure of educational and psychological assessments. However, CFA is often applied without an explicit validity argument or is limited to evaluating a single hypothesized measurement model. Consequently, validity conclusions may become fragmented and data-driven rather than based on a systematic evaluation of alternative explanations for score meaning. Such approaches may overlook both the causal interpretation implicit in the common factor model and plausible alternative explanations for variation in item responses. This thesis reconceptualizes the role of CFA within Kane’s argument-based approach to validation by using CFA to evaluate competing causal validity arguments. Each argument specifies assumptions about the constructs or other factors hypothesized to cause variation in item responses and is represented by a prespecified CFA model. The proposed approach is demonstrated using the Canadian PIRLS 2016 reading assessment. Three causal validity arguments, corresponding to three intended score interpretations—holistic, two-purpose-based, and two-process-based—were formulated according to the assessment blueprint. For each intended argument, a corresponding counter-validity argument proposed that item responses were partly attributable to passage-specific testlet effects. Evidence from the six CFA models was integrated to evaluate the intended score interpretations and their competing explanations. The two-purpose-based score interpretation received the strongest support; the holistic score interpretation was challenged by evidence of passage-specific testlet effects, and the two-process-based score interpretation received the least support. The case study demonstrates three implications for validation: validation should focus on each intended score interpretation rather than on the test as a whole; plausible competing validity arguments should be considered; and both favorable and unfavorable CFA estimation behavior and model-fit results can provide validity evidence when interpreted in relation to the argument being evaluated.