Comparing the Weighted Gain Score and a Rasch-Based Approach for Estimating Learning Outcomes in Medical Education: Quantitative Study

Background: Pretest-posttest designs are widely used to estimate learning gain in studies evaluating educational interventions in medical education. The Weighted Gain Score (WGS) was proposed to reduce bias associated with differences in baseline performance. Objective: This study evaluated the statistical and inferential properties of the WGS by comparing it to Rasch Learning Gain (RLG) across 3 datasets. Methods: The WGS implements a weighting coefficient that includes the parameter µ, which linearly rescales the difference between pretest and posttest percentage scores. We examined the effect of varying µ (30, 50, and 70) on learning gain calculations and compared the results with those obtained using RLG. The following three datasets were analyzed: (1) a small illustrative dataset demonstrating the mathematical behavior of the WGS, (2) an empirical dataset from a previous educational evaluation study, and (3) a randomly generated binomial dataset designed to examine the metric under larger sample conditions. Results: Changing the parameter µ in the WGS affected the magnitude of the calculated learning gains: lower µ-values produced larger gain estimates, whereas higher µ-values produced smaller estimates. Despite these differences in scale, the WGS and RLG correlated strongly in both the empirical dataset (r=0.93; P<.001) and the simulated dataset (r=0.92; P<.001); variation in µ did not alter the inferential results. Both methods identified the same interaction effect in the empirical dataset. Conclusions: The WGS produced results highly consistent with those of RLG while requiring substantially lower computational complexity. The metric can be applied to both small and large datasets and allows µ to function as an adjustment coefficient for calibrating learning gain estimates across cohorts without altering inferential conclusions.