Growth Bias and False-Positive Learning Claims in AI-Assisted L2 Writing Assessment
AI-supported writing gains may combine durable independent proficiency, immediate assistance, and changes in measurement conditions. This fully synthetic Monte Carlo study compared six estimators under four primary scenarios (2,000 replications per scenario; n = 600 per replication), expanded sensitivity scenarios (1,200 replications each), and independent missing-data validation. A further 1,200-replication analysis per response mechanism evaluated fitted inverse-probability weighting (IPW) with stacked-sandwich uncertainty; paired-mode calibration was stress-tested under assessment-period effects, directional carryover, and task-by-mode interactions. Under zero true AI-related learning and a mean direct assistance effect of 0.50 standardised score units, assisted ANCOVA estimated the supported-outcome effect as 0.498 and yielded a statistically positive result in every primary replication when mistakenly interpreted as autonomous learning; it was not biased for its legitimate assisted-outcome target. Representative paired calibration reduced endpoint bias to −0.003 (94.7% coverage). In the new interval experiment, fitted-IPW stacked-sandwich coverage for the true unaided-transfer effect was 94.0% under MCAR, 93.8% under MAR, and 24.7% under MNAR; corresponding MNAR effect bias was −0.153. With balanced assessment order, a stipulated 0.20-unit carryover effect still produced +0.103 calibration bias. Conclusions: Supported performance, independent endpoint learning, and subsequent unaided transfer are distinct targets. Valid inference depends on construct alignment, measurement transportability, observability, selection assumptions, and appropriate uncertainty estimation. All effects and operating characteristics are conditional on synthetic data-generating mechanisms, not observed AI effects in classrooms.
Authors
- Tan Nguyen Huynh Si (ORCID: https://orcid.org/0009-0000-9801-2760)
Institutions
- University of Phan Thiet (VN)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-18
- DOI
- https://doi.org/10.21203/rs.3.rs-11765598/v1
- Primary Topic
- Advanced Causal Inference Techniques
- Type
- preprint