Vanity metrics but no real gains the role of generative AI in unsupervised assessments in finance

The emergence of generative artificial intelligence (GAI) chatbots has raised concerns about the validity of unsupervised online assessments in higher education. This paper examines the impact of GAI on student performance by comparing outcomes in unsupervised online quizzes and supervised written exams before and after the widespread access to GAI, using data from a large-enrollment Finance course spanning eight cohorts (2021–2024) and more than 75,000 observations. By means of a differences-in-differences design, we find a substantial increase in performance on unsupervised assessments following GAI’s arrival, while no corresponding gains are observed in supervised exams, suggesting that the performance improvements reflect GAI-assistance rather than genuine learning. Consistent with GAI collapsing the search costs that previously made conceptual material hard to look up, the effect is driven almost entirely by theoretical questions, a stark reversal from the pre-GAI period, when students underperformed on such questions in unsupervised settings. Exploratory analyses show that the effect is similar across student ability groups, except for the highest-performing students, who exhibit only limited gains; that there is no evidence of gender differences; and that the effect grows across post-GAI cohorts, consistent with expanded adoption and improving tools.

Authors

Publication Details

Journal
Figshare
Published
2026-11-17
DOI
https://doi.org/10.6084/m9.figshare.35040188
Primary Topic
Artificial Intelligence in Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Vanity metrics but no real gains the role of generative AI in unsupervised assessments in finance

Iñaki Rodríguez-Longarela, Cormac McGrath
2 links
Figshare
Artificial Intelligence in Education
article

Vanity metrics but no real gains the role of generative AI in unsupervised assessments in finance

Iñaki Rodríguez-Longarela, Cormac McGrath
article en

Abstract

The emergence of generative artificial intelligence (GAI) chatbots has raised concerns about the validity of unsupervised online assessments in higher education. This paper examines the impact of GAI on student performance by comparing outcomes in unsupervised online quizzes and supervised written exams before and after the widespread access to GAI, using data from a large-enrollment Finance course spanning eight cohorts (2021–2024) and more than 75,000 observations. By means of a differences-in-differences design, we find a substantial increase in performance on unsupervised assessments following GAI’s arrival, while no corresponding gains are observed in supervised exams, suggesting that the performance improvements reflect GAI-assistance rather than genuine learning. Consistent with GAI collapsing the search costs that previously made conceptual material hard to look up, the effect is driven almost entirely by theoretical questions, a stark reversal from the pre-GAI period, when students underperformed on such questions in unsupervised settings. Exploratory analyses show that the effect is similar across student ability groups, except for the highest-performing students, who exhibit only limited gains; that there is no evidence of gender differences; and that the effect grows across post-GAI cohorts, consistent with expanded adoption and improving tools.

Figshare
Quality education
Artificial Intelligence in Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon