Independent Learning after Generative AI Support in School Mathematics: A Systematic Review and Assessment-Stratified Meta-Analysis

Background and Objective. Generative artificial intelligence (GenAI) solves school math problems on demand. Higher scores during live tool access do not prove retained mathematical competence once that support disappears. Earlier meta-analyses pooled assisted task completion together with unmonitored and closed-book posttests, which obscured whether pupils actually learned to work alone. This PRISMA 2020 systematic review and stratified meta-analysis evaluates K–12 mathematics outcomes by separating test-time resource rules from instructional design. Methods. Two-stream searches across ERIC, APA PsycNet, OpenAlex, Crossref, Semantic Scholar, arXiv, EuropePMC, and bidirectional citation chasing (1 January 2022–8 November 2026) returned 11,061 raw entries and 6,530 unique records. Every sought report was retrieved in full (372 / 372; 0 unretrieved). Eligibility verification and mathematical consistency checks retained 49 K–12 school-mathematics reports across 46 independent cohorts and supplied 36 quantitative contrasts across 20 controlled study families (21 controlled quantitative reports, plus 1 factorial within-AI modality trial synthesized narratively) alongside 28 qualitative, single-arm, observational, and boundary-comparator reports. Random-effects Paule–Mandel and restricted maximum likelihood estimators with classroom cluster adjustments (calibrated intraclass correlation ρ = 0.128) stratified contrasts across Category A (GenAI active during testing), Category B (posttest with undocumented resource rules), Category C (partial human-help restriction), and Category D (explicit closedresource immediate and delayed exams). Results. Within primary multi-cluster and individually randomized trials (k = 14), large concurrent gains during live GenAI access (Category A: g = +0.615, 95% CI [+0.044, +1.185], p = .035) shrank across undocumented posttests (Category B: g = +0.186 [+0.009, +0.362]) and tutor-restricted exit tickets (Category C: g = +0.095 [+0.024, +0.167]) down to a tight null on explicit closed-resource exams (Category D: g = −0.010 [−0.137, +0.117], p = .876; focal contrast Δg = +0.625, p = .036). Extended controlled trials (k = 28) confirmed strong between-category moderation (Q_between = 12.55, p = .0059; Category D: g = −0.035 [−0.149, +0.078]). Pedagogical architecture accounted for all primary post-withdrawal heterogeneity (Q_between = 18.69, p = .0001; I² = 0% within every tier). Unconstrained direct-answer access harmed closed-exam scores (M1: g = −0.205, p = .043), passive hint guardrails blocked harm without yielding net transfer (M2: g = −0.038, p = .173), and architectures enforcing productive cognitive friction (unaided mastery gating, teachable agents, Socratic Pólya scaffolding, and human-in-the-loop oversight) produced durable gains (M3: g = +0.100, p = .0001; extended g = +0.315, p = .0002). Head-to-head within-AI trials (k = 6) favored pedagogical guardrails and teacher mediation over unguarded GenAI (g = +0.645 [+0.215, +1.075], p = .0033), whereas upper-secondary algebra and calculus cohorts showed acute offloading and workingmemory overload. Conclusions. Unstratified, unclustered synthesis inflates verified closed-resource mathematics learning by +0.26 to +0.29 SD (naive g = +0.275 vs. closed-resource g = −0.010). Durable competence requires crossing two distinct thresholds. Instructional systems must block direct answer copying and enforce active learner generation before and after AI withdrawal.

Authors

Publication Details

Journal
Research Square
Published
2026-11-19
DOI
https://doi.org/10.21203/rs.3.rs-11766988/v1
Primary Topic
Artificial Intelligence in Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Independent Learning after Generative AI Support in School Mathematics: A Systematic Review and Assessment-Stratified Meta-Analysis

Alexander Kuznetsov
Research Square
Artificial Intelligence in Education
preprint

Independent Learning after Generative AI Support in School Mathematics: A Systematic Review and Assessment-Stratified Meta-Analysis

Alexander Kuznetsov
preprint en

Abstract

Background and Objective. Generative artificial intelligence (GenAI) solves school math problems on demand. Higher scores during live tool access do not prove retained mathematical competence once that support disappears. Earlier meta-analyses pooled assisted task completion together with unmonitored and closed-book posttests, which obscured whether pupils actually learned to work alone. This PRISMA 2020 systematic review and stratified meta-analysis evaluates K–12 mathematics outcomes by separating test-time resource rules from instructional design. Methods. Two-stream searches across ERIC, APA PsycNet, OpenAlex, Crossref, Semantic Scholar, arXiv, EuropePMC, and bidirectional citation chasing (1 January 2022–8 November 2026) returned 11,061 raw entries and 6,530 unique records. Every sought report was retrieved in full (372 / 372; 0 unretrieved). Eligibility verification and mathematical consistency checks retained 49 K–12 school-mathematics reports across 46 independent cohorts and supplied 36 quantitative contrasts across 20 controlled study families (21 controlled quantitative reports, plus 1 factorial within-AI modality trial synthesized narratively) alongside 28 qualitative, single-arm, observational, and boundary-comparator reports. Random-effects Paule–Mandel and restricted maximum likelihood estimators with classroom cluster adjustments (calibrated intraclass correlation ρ = 0.128) stratified contrasts across Category A (GenAI active during testing), Category B (posttest with undocumented resource rules), Category C (partial human-help restriction), and Category D (explicit closedresource immediate and delayed exams). Results. Within primary multi-cluster and individually randomized trials (k = 14), large concurrent gains during live GenAI access (Category A: g = +0.615, 95% CI [+0.044, +1.185], p = .035) shrank across undocumented posttests (Category B: g = +0.186 [+0.009, +0.362]) and tutor-restricted exit tickets (Category C: g = +0.095 [+0.024, +0.167]) down to a tight null on explicit closed-resource exams (Category D: g = −0.010 [−0.137, +0.117], p = .876; focal contrast Δg = +0.625, p = .036). Extended controlled trials (k = 28) confirmed strong between-category moderation (Q_between = 12.55, p = .0059; Category D: g = −0.035 [−0.149, +0.078]). Pedagogical architecture accounted for all primary post-withdrawal heterogeneity (Q_between = 18.69, p = .0001; I² = 0% within every tier). Unconstrained direct-answer access harmed closed-exam scores (M1: g = −0.205, p = .043), passive hint guardrails blocked harm without yielding net transfer (M2: g = −0.038, p = .173), and architectures enforcing productive cognitive friction (unaided mastery gating, teachable agents, Socratic Pólya scaffolding, and human-in-the-loop oversight) produced durable gains (M3: g = +0.100, p = .0001; extended g = +0.315, p = .0002). Head-to-head within-AI trials (k = 6) favored pedagogical guardrails and teacher mediation over unguarded GenAI (g = +0.645 [+0.215, +1.075], p = .0033), whereas upper-secondary algebra and calculus cohorts showed acute offloading and workingmemory overload. Conclusions. Unstratified, unclustered synthesis inflates verified closed-resource mathematics learning by +0.26 to +0.29 SD (naive g = +0.275 vs. closed-resource g = −0.010). Durable competence requires crossing two distinct thresholds. Instructional systems must block direct answer copying and enforce active learner generation before and after AI withdrawal.

Research Square
Artificial Intelligence in Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon