Is the Treatment Weak or the Test Insensitive? Interrogating the Nature of Treatment Effects on Standardized Tests

ABSTRACT A general consensus exists that disabilities are harder to treat in older students and improvements are harder to demonstrate on standardized tests. This study used advanced psychometric modeling and computer simulation to examine factors affecting the sensitivity of the Gates‐MacGinitie Reading Test to intervention effects in low ability, middle school students and to examine the possibility that off‐level testing may increase sensitivity to treatment effects. We examine why test sensitivity may vary with student ability and test form used for selection before using computer simulation to examine sensitivity to 10 patterns and sizes of treatment effects. Off‐level testing offered only small improvements to sensitivity. At the same time, findings raised concerns that the lack of effects in real treatment studies may stem from treatments whose effects are either weak or inconsistent and highlighted the failure of intervention research to comprehensively consider psychometric characteristics of measures in designing treatment studies.