Students-Generative AI interaction patterns and its impact on academic writing

Abstract Considering both the transformative opportunities and challenges presented by generative AI (GenAI) in academic writing, effectively integrating GenAI into the academic setting becomes a significant need requiring prioritization. Yet, there is limited understanding regarding the nature of interactions between different types of students, what behavioral patterns students exhibit during a student-GenAI interaction (SAI) on a given task, and how these different SAI patterns relate to the actual writing task performance. This study, therefore, aimed to identify SAI patterns of academic writing tasks depending on students’ level of AI literacy and examine the differences in academic writing performance between the identified SAI patterns. Drawing from the combination of three data sources, including think-aloud protocols, screen-recordings, and chat histories between 36 Chinese graduate students and a GenAI writing system, epistemic network analysis (ENA) was used to reveal the distinctive SAI patterns of students with different levels of AI literacy. The study found that students with a high level of AI literacy exhibited a collaborative approach to SAI, actively accepting GenAI’s suggestions and engaging GenAI in meta-cognitive-related activities such as planning, whereas students with a low level of AI literacy demonstrated much less interaction with GenAI in completing their writing tasks, instead choosing to ideate and evaluate independently. In addition, the Wilcoxon rank-sum (Mann-Whitney U) test was conducted to assess the writing task performance of the two AI literacy groups. Findings revealed statistical differences in all evaluation rubrics (content, structure/organization, expression). This study offers implications for the design and implementation of GenAI agents in writing tasks and the pedagogy of GenAI-assisted instruction.