Tracing Student Thinking in Socratic LLM Dialogues: A Reflexive Thematic Analysis

This exploratory qualitative study examines how students’ reasoning becomes observable and develops within Socratic dialogues with Riffbot, a large language model (LLM)-based learning chatbot designed to guide students from descriptive and clarificatory questioning toward explanatory and reflective inquiry. The empirical dataset comprised 121 dialogue transcripts generated by 93 students across seven project-based courses at a university of applied sciences. All 121 transcripts were initially reviewed to examine participation, dialogue length, and interactional characteristics. Of these, 72 dialogues containing at least 10 turns were selected for detailed coding and reflexive thematic analysis (RTA), while 49 shorter dialogues were retained as contextual material. The analysis examined student responses in relation to successive chatbot prompts, using theoretically informed reasoning categories to trace interactional patterns within complete dialogues. Coded data were visualized in a chronological matrix to support the interpretation of reasoning trajectories. Descriptive reasoning predominated, while some dialogues showed emerging explanatory and reflective reasoning. Other trajectories remained primarily descriptive, involved repeated clarification, or moved recursively between reasoning categories, often including moments of uncertainty and redirection. These findings highlight the non-linear and interactionally situated nature of learning expressed during Socratic LLM–student exchanges, without establishing independent learning gains. Methodologically, the study demonstrates how systematic coding, chronological visualization, and reflexive thematic interpretation can be combined to make thinking-in-action more visible and analytically traceable. It contributes an empirical account of student reasoning within AI-mediated dialogue and a practical, digitally supported approach to investigating learning-related processes in higher education.