Growth Bias and False-Positive Learning Claims in AI-Assisted L2 Writing Assessment

AI-supported writing gains may combine durable independent proficiency, immediate assistance, and changes in measurement conditions. This fully synthetic Monte Carlo study compared six estimators under four primary scenarios (2,000 replications per scenario; n = 600 per replication), expanded sensitivity scenarios (1,200 replications each), and independent missing-data validation. A further 1,200-replication analysis per response mechanism evaluated fitted inverse-probability weighting (IPW) with stacked-sandwich uncertainty; paired-mode calibration was stress-tested under assessment-period effects, directional carryover, and task-by-mode interactions. Under zero true AI-related learning and a mean direct assistance effect of 0.50 standardised score units, assisted ANCOVA estimated the supported-outcome effect as 0.498 and yielded a statistically positive result in every primary replication when mistakenly interpreted as autonomous learning; it was not biased for its legitimate assisted-outcome target. Representative paired calibration reduced endpoint bias to −0.003 (94.7% coverage). In the new interval experiment, fitted-IPW stacked-sandwich coverage for the true unaided-transfer effect was 94.0% under MCAR, 93.8% under MAR, and 24.7% under MNAR; corresponding MNAR effect bias was −0.153. With balanced assessment order, a stipulated 0.20-unit carryover effect still produced +0.103 calibration bias. Conclusions: Supported performance, independent endpoint learning, and subsequent unaided transfer are distinct targets. Valid inference depends on construct alignment, measurement transportability, observability, selection assumptions, and appropriate uncertainty estimation. All effects and operating characteristics are conditional on synthetic data-generating mechanisms, not observed AI effects in classrooms.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-18
DOI
https://doi.org/10.21203/rs.3.rs-11765598/v1
Primary Topic
Advanced Causal Inference Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Growth Bias and False-Positive Learning Claims in AI-Assisted L2 Writing Assessment

Tan Nguyen Huynh Si
1 links
Research Square
Advanced Causal Inference Techniques
preprint

Growth Bias and False-Positive Learning Claims in AI-Assisted L2 Writing Assessment

Tan Nguyen Huynh Si
preprint en

Abstract

AI-supported writing gains may combine durable independent proficiency, immediate assistance, and changes in measurement conditions. This fully synthetic Monte Carlo study compared six estimators under four primary scenarios (2,000 replications per scenario; n = 600 per replication), expanded sensitivity scenarios (1,200 replications each), and independent missing-data validation. A further 1,200-replication analysis per response mechanism evaluated fitted inverse-probability weighting (IPW) with stacked-sandwich uncertainty; paired-mode calibration was stress-tested under assessment-period effects, directional carryover, and task-by-mode interactions. Under zero true AI-related learning and a mean direct assistance effect of 0.50 standardised score units, assisted ANCOVA estimated the supported-outcome effect as 0.498 and yielded a statistically positive result in every primary replication when mistakenly interpreted as autonomous learning; it was not biased for its legitimate assisted-outcome target. Representative paired calibration reduced endpoint bias to −0.003 (94.7% coverage). In the new interval experiment, fitted-IPW stacked-sandwich coverage for the true unaided-transfer effect was 94.0% under MCAR, 93.8% under MAR, and 24.7% under MNAR; corresponding MNAR effect bias was −0.153. With balanced assessment order, a stipulated 0.20-unit carryover effect still produced +0.103 calibration bias. Conclusions: Supported performance, independent endpoint learning, and subsequent unaided transfer are distinct targets. Valid inference depends on construct alignment, measurement transportability, observability, selection assumptions, and appropriate uncertainty estimation. All effects and operating characteristics are conditional on synthetic data-generating mechanisms, not observed AI effects in classrooms.

Research Square
University of Phan Thiet (VN)
Quality education
Advanced Causal Inference Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon