Reconsidering the utility of mean comparisons in evaluative work

Abstract Objective The purpose was to evaluate the appropriateness of mean comparisons for evaluation work by testing the measurement equivalence of traditional pretests, retrospective pretests, and posttests. Background Scholars have debated the use of a retrospective pretest as a viable method for evaluating interventions. Although they disagree on the origins of the difference in means between traditional pretests and retrospective pretests, both sides note a change in how participants perceive the construct. Which, if either, version of the pretest is consistent with perceptions on posttest response can be tested using measurement equivalence. Method One hundred twelve participants from a relationship education intervention took a traditional pretest, retrospective pretest, and posttest on two targeted outcomes, relationship confidence and relationship knowledge. A series of measurement equivalence tests were conducted between the traditional pretest, retrospective pretest, and posttest. Equivalence was tested by comparing nested confirmatory factor models. Results For both outcomes, the traditional pretest failed to demonstrate equivalence with the posttest. The retrospective pretest, on the other hand, demonstrated strong equivalence with the posttest. Conclusion For our sample and measure, mean comparisons between traditional pretests and posttests are not appropriate, as the nature of the measurement has changed. The retrospective pretest is an appropriate choice, and its use allows for comparison against the posttest mean. Implications Traditional pretest–posttest designs may be inappropriate for evaluating interventions that potentially change the constructs being measured; alternative formats, such as retrospective pretests or qualitative work may be needed. When planning and preparing the evaluation of an intervention, potential changes in perception of the outcome (and associated measurement concerns, such as equivalence) should be considered and accounted for.