Challenges in using online assessments to measure student understanding in the era of generative AI

Many courses now incorporate online methods for assessing student learning. Some instructors assign questions prior to lecture (e.g., prelecture quizzes) to obtain information that helps them tailor their lessons to the current state of student understanding. To assess curriculum efficacy, physics education researchers often use “pretests,” inventories, etc. that provide baseline data that can be compared to results from post-tests. These assessments were initially given on paper but moved online in the early 2000s. At the University of Washington, we found that the impact of this change was minimal; the results of our pretests stayed essentially the same when given on paper or online and were consistent from quarter to quarter and instructor to instructor. However, we have begun to observe systematic changes in pretest performance that are consistent with students using artificial intelligence (AI) tools to answer pretests, even though they are scored only on effort, not correctness. Using two-proportion z -tests, mixed-effects logistic regression, and Welch’s t tests to analyze response length of explanations, this paper documents the nature and extent of these changes across 39 questions in two introductory algebra-based physics courses. This paper presents results that suggest caution to those using online quizzes to probe student understanding—both before and/or after instruction. We have seen increases in correct responses on pretests as great as 36%, but we have also seen significant decreases on questions that AI tools have tended to answer incorrectly. These findings raise important questions about the validity of online assessments and are intended to raise awareness for anyone using online tests to assess student learning. It remains an open question as to whether, and how, AI can be leveraged to improve instruction.

Authors

Institutions

Publication Details

Journal
Physical Review Physics Education Research
Published
2026-11-19
DOI
https://doi.org/10.1103/xrw4-t58t
Primary Topic
Academic integrity and plagiarism
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Challenges in using online assessments to measure student understanding in the era of generative AI

Paula R. L. Heron, Peter S. Shaffer, David P. Smith
5 links
Physical Review Physics Education Research
Academic integrity and plagiarism
article

Challenges in using online assessments to measure student understanding in the era of generative AI

Paula R. L. Heron, Peter S. Shaffer, David P. Smith
article en

Abstract

Many courses now incorporate online methods for assessing student learning. Some instructors assign questions prior to lecture (e.g., prelecture quizzes) to obtain information that helps them tailor their lessons to the current state of student understanding. To assess curriculum efficacy, physics education researchers often use “pretests,” inventories, etc. that provide baseline data that can be compared to results from post-tests. These assessments were initially given on paper but moved online in the early 2000s. At the University of Washington, we found that the impact of this change was minimal; the results of our pretests stayed essentially the same when given on paper or online and were consistent from quarter to quarter and instructor to instructor. However, we have begun to observe systematic changes in pretest performance that are consistent with students using artificial intelligence (AI) tools to answer pretests, even though they are scored only on effort, not correctness. Using two-proportion z -tests, mixed-effects logistic regression, and Welch’s t tests to analyze response length of explanations, this paper documents the nature and extent of these changes across 39 questions in two introductory algebra-based physics courses. This paper presents results that suggest caution to those using online quizzes to probe student understanding—both before and/or after instruction. We have seen increases in correct responses on pretests as great as 36%, but we have also seen significant decreases on questions that AI tools have tended to answer incorrectly. These findings raise important questions about the validity of online assessments and are intended to raise awareness for anyone using online tests to assess student learning. It remains an open question as to whether, and how, AI can be leveraged to improve instruction.

Physical Review Physics Education ResearchVol. 22(2)
University of Washington (US)
Quality education
Academic integrity and plagiarism
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon