Can the Field Tell? A Reporting Audit of Whether AI-Assisted Learning Studies Say If the AI Was Present When Learning Was Measured

A study that says an AI tutor improved learning needs to say one more thing: was the AI still there when learning was measured? If it was, the score describes the tool and the student together. This audit asks a narrower question than "does AI help": can a reader tell, from the paper, which situation applied? The corpus is the union of the primary studies behind three recent meta-analyses of AI-assisted learning, 107 studies. I retrieved 83 full texts by a fixed cutoff and coded one outcome measure per study against a frozen coding manual. Three language models, called by fixed identifiers, coded the 63 non-pilot papers with a performance outcome; every quotation they returned was string-checked against the paper, and all 378 passed. Two human coders coded the ten pilot papers and agreed 10 of 10 on the primary variable. One human coded a seeded random sample of 15 papers without opening the model answers. Across the 73 performance-outcome papers, 51 to 56 (69.9% to 76.7%) do not state whether the AI was available during the outcome assessment; the range covers five papers the models split on. Over all 107 studies, counting the 24 not retrieved, the range is 51 to 80 (47.7% to 74.8%). Of the 16 papers that do say, 11 had the tool available and 5 did not. I propose a six-item reporting checklist and one design rule: every learning-effect study includes at least one outcome measured with the AI absent, and says so. The audit was not preregistered, the extraction was model-assisted, and I sell a measurement product in this area. Each limit is set out in the paper, and every number can be recomputed from the per-call records.

Authors

Publication Details

Published
2026-10-27
DOI
https://doi.org/10.35542/osf.io/kpnrq_v1
Primary Topic
Intelligent Tutoring Systems and Adaptive Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Can the Field Tell? A Reporting Audit of Whether AI-Assisted Learning Studies Say If the AI Was Present When Learning Was Measured

Kumar Sumbhav Srivastava
4 links
Intelligent Tutoring Systems and Adaptive Learning
preprint

Can the Field Tell? A Reporting Audit of Whether AI-Assisted Learning Studies Say If the AI Was Present When Learning Was Measured

Kumar Sumbhav Srivastava
preprint en

Abstract

A study that says an AI tutor improved learning needs to say one more thing: was the AI still there when learning was measured? If it was, the score describes the tool and the student together. This audit asks a narrower question than "does AI help": can a reader tell, from the paper, which situation applied? The corpus is the union of the primary studies behind three recent meta-analyses of AI-assisted learning, 107 studies. I retrieved 83 full texts by a fixed cutoff and coded one outcome measure per study against a frozen coding manual. Three language models, called by fixed identifiers, coded the 63 non-pilot papers with a performance outcome; every quotation they returned was string-checked against the paper, and all 378 passed. Two human coders coded the ten pilot papers and agreed 10 of 10 on the primary variable. One human coded a seeded random sample of 15 papers without opening the model answers. Across the 73 performance-outcome papers, 51 to 56 (69.9% to 76.7%) do not state whether the AI was available during the outcome assessment; the range covers five papers the models split on. Over all 107 studies, counting the 24 not retrieved, the range is 51 to 80 (47.7% to 74.8%). Of the 16 papers that do say, 11 had the tool available and 5 did not. I propose a six-item reporting checklist and one design rule: every learning-effect study includes at least one outcome measured with the AI absent, and says so. The audit was not preregistered, the extraction was model-assisted, and I sell a measurement product in this area. Each limit is set out in the paper, and every number can be recomputed from the per-call records.

Quality education
Intelligent Tutoring Systems and Adaptive Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Can the Field Tell? A Reporting Audit of Whether AI-Assisted Learning Studies Say If the AI Was Present When Learning Was Measured — Kumar Sumbhav Srivastava · (2026) | TGRS Research Map | TGRS