Exploring analytic flexibility in model fit evaluation : a multiverse analysis applied to measurement invariance

Measurement invariance (MI) testing evaluates whether measures function equivalently across groups, but MI conclusions depend on both how model fit is computed and how researchers translate fit evidence into decisions. This thesis examines analytic flexibility in multi-group confirmatory factor analysis based MI testing across two studies. In Study 1, we reviewed 101 peer-reviewed empirical papers published in 2023 and indexed in APA PsycNet that used MGCFA for MI testing. We coded reported fit indices, cutoffs, and decision rules, as well as related analytic choices. Reporting practices were highly heterogeneous: we identified 49 distinct global fit rules for configural invariance and 37 and 33 distinct change-in-fit rules for the metric/threshold and scalar steps, respectively. No single full decision rule dominated, and some papers did not clearly report criteria. In Study 2, we used the codable decision rules from Study 1 in a multiverse analysis of an open VVIQ-2 dataset, holding the data, preprocessing, and measurement model fixed while varying estimator family, fit index computation variant, and decision rule. Results showed that invariance decisions were sensitive to these choices. Configural pass rates ranged from 20.8% to 83.3% across estimator and variant workflows, and later-step pass rates also varied, especially for rules requiring nonsignificant Δχ² tests or stringent ΔCFI cutoffs. These results should be interpreted as a proof of concept rather than as a general claim about the VVIQ-2. They show that MI conclusions can depend on defensible fit evaluation workflows. We recommend that MI reports state the estimator, software and version, response scale treatment, fit index variants, difference tests, and full decision logic, and that researchers use sensitivity analyses to evaluate whether substantive cross group conclusions are robust to analytic choices.