Performance Is Not Learning: Designing courses for the age of AI, and four principles from the evidence

Generative AI has made one of the oldest distinctions in learning science newly urgent: performance during practice is not the same as learning that lasts. In a randomized field experiment, students practicing with a ChatGPT-style assistant scored 48% higher during practice and 17% lower once the assistant was removed. In a randomized trial at Harvard, a structured AI tutor more than doubled median learning gains over an in-class active-learning lesson. The technology was comparable; the design was not. This article reads those trials alongside course-level studies of optional guardrails, oral code reviews, and the link between confidence and critical thinking. It proposes four design principles for courses where AI is present, considers what they mean for higher education institutions, and states the limits of the evidence.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24204020
Primary Topic
Artificial Intelligence in Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Performance Is Not Learning: Designing courses for the age of AI, and four principles from the evidence

Clark Ngo
3 links
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Education
preprint

Performance Is Not Learning: Designing courses for the age of AI, and four principles from the evidence

Clark Ngo
preprint en

Abstract

Generative AI has made one of the oldest distinctions in learning science newly urgent: performance during practice is not the same as learning that lasts. In a randomized field experiment, students practicing with a ChatGPT-style assistant scored 48% higher during practice and 17% lower once the assistant was removed. In a randomized trial at Harvard, a structured AI tutor more than doubled median learning gains over an in-class active-learning lesson. The technology was comparable; the design was not. This article reads those trials alongside course-level studies of optional guardrails, oral code reviews, and the link between confidence and critical thinking. It proposes four design principles for courses where AI is present, considers what they mean for higher education institutions, and states the limits of the evidence.

Zenodo (CERN European Organization for Nuclear Research)
City University of Seattle (US)
Quality education
Artificial Intelligence in Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon