From RAMSing to Rigor: Improving Moderator Analysis in STEM Education Meta-Research

Abstract Moderators are intended to clarify when, where, and for whom interventions work, yet null moderator results are often omitted from the published record. We term this practice “reporting after moderators are significant” (RAMSing) and examine its scope and consequences in STEM education meta-research. Drawing on an umbrella review of 89 meta-analyses comprising 1,786 moderator effect sizes, we found a substantial visibility gap: non-significant moderator effect sizes showed markedly lower relative visibility than the most strongly selected results ( $$\:{\upomega\:}$$ $$\:\approx\:$$ 0.40). To address this bias, we combined structural mapping with two bias-robust Bayesian frameworks, RoBMA-PSMA and RoBMA-Regression, to estimate bias-adjusted moderator effects, derive field-specific benchmarks, and estimate realistic evidential requirements for moderator testing. The overall bias-adjusted mean moderator effect was g = 0.27 (95% CrI [0.20, 0.34]) with substantial heterogeneity ( $$\:{\uptau\:}$$ = 0.48). Modeling a five-level moderator-theme classification substantially improved fit (BF Inc = 350) and slightly reduced residual heterogeneity ( $$\:{\uptau\:}$$ = 0.44). RAMSing-adjusted benchmarks placed the 25th, 50th, and 75th percentiles of | g | at 0.18, 0.38, and 0.64, respectively, all below both Cohen’s conventional heuristics and the raw empirical quartiles. Prospective power analyses further indicated that, under the assumptions examined here ( g = 0.38, $$\:{{\uptau\:}}^{2}$$ = 0.24, $$\:{\upalpha\:}$$ = 0.05), approximately 14 independent effect sizes per moderator level are needed to detect a typical moderator effect. These findings indicate that RAMSing is likely an important source of distortion in moderator-based inference within this corpus. The workflow and openly available toolkit introduced here provide a practical framework for detecting and bias-adjusting RAMSing, and for supporting practices designed to reduce it in STEM education and other fields that rely on meta-analytic moderator evidence.