What Makes Learning with GenAI Effective? Text Support and Deeper Engagement are Associated with Higher Performance in an Ecology Context

Abstract Generative artificial intelligence (GenAI) tools such as ChatGPT are increasingly used for learning, yet evidence remains limited regarding their effectiveness in science learning contexts and how learner–AI interaction relates to performance. In a randomized experiment, 185 university students worked on task sheets about lake eutrophication and restoration. Depending on condition, they could use two informational texts (text-only), ChatGPT configured to answer only on the basis of these texts (ChatGPT-only), or both (text + ChatGPT). Afterwards, all students completed a knowledge test without access to any materials. Test performance was highest in the text-only and text + ChatGPT conditions and lowest in the ChatGPT-only condition, yielding a robust overall condition effect. Chat logs from the ChatGPT conditions were analyzed to examine interaction patterns associated with performance. Although ChatGPT-only students produced substantially more interaction turns than students with access to both texts and ChatGPT, interaction frequency was unrelated to test performance. Instead, specific interaction patterns were differentially associated with performance: deepening questions were positively associated, whereas direct acceptance of AI-generated responses was negatively associated with test performance. Output-steering and questioning additionally emerged as positively associated with performance in individual analyses. Prompting behavior varied with task difficulty, with deepening questions increasing for more demanding ecology tasks. The findings suggest that, in science learning contexts, the educational value of GenAI may depend less on access to the technology than on learners’ epistemic engagement with it.

Authors

Institutions

Publication Details

Journal
Journal of Science Education and Technology
Published
2026-11-12
DOI
https://doi.org/10.1007/s10956-026-10374-w
Primary Topic
Artificial Intelligence in Education
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

What Makes Learning with GenAI Effective? Text Support and Deeper Engagement are Associated with Higher Performance in an Ecology Context

Detlef Urhahne, Lars Dietrich, Kerstin Kremer, Laura Kehle
Journal of Science Education and Technology
Artificial Intelligence in Education
article

What Makes Learning with GenAI Effective? Text Support and Deeper Engagement are Associated with Higher Performance in an Ecology Context

Detlef Urhahne, Lars Dietrich, Kerstin Kremer, Laura Kehle
article en

Abstract

Abstract Generative artificial intelligence (GenAI) tools such as ChatGPT are increasingly used for learning, yet evidence remains limited regarding their effectiveness in science learning contexts and how learner–AI interaction relates to performance. In a randomized experiment, 185 university students worked on task sheets about lake eutrophication and restoration. Depending on condition, they could use two informational texts (text-only), ChatGPT configured to answer only on the basis of these texts (ChatGPT-only), or both (text + ChatGPT). Afterwards, all students completed a knowledge test without access to any materials. Test performance was highest in the text-only and text + ChatGPT conditions and lowest in the ChatGPT-only condition, yielding a robust overall condition effect. Chat logs from the ChatGPT conditions were analyzed to examine interaction patterns associated with performance. Although ChatGPT-only students produced substantially more interaction turns than students with access to both texts and ChatGPT, interaction frequency was unrelated to test performance. Instead, specific interaction patterns were differentially associated with performance: deepening questions were positively associated, whereas direct acceptance of AI-generated responses was negatively associated with test performance. Output-steering and questioning additionally emerged as positively associated with performance in individual analyses. Prompting behavior varied with task difficulty, with deepening questions increasing for more demanding ecology tasks. The findings suggest that, in science learning contexts, the educational value of GenAI may depend less on access to the technology than on learners’ epistemic engagement with it.

Journal of Science Education and Technology
University of Passau (DE), Justus-Liebig-Universität Gießen (DE)
Universität Passau
Quality education
Artificial Intelligence in Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon