Using large language models to analyze student reflections on generative AI in graduate coursework

This exploratory study examines graduate students’ perceptions of generative AI use in coursework and evaluates the feasibility of using large language models (LLMs) to support thematic coding of brief open-ended responses. Students ( n = 20) enrolled in a systems analysis and design course completed a post-course survey consisting of Likert-scale items and eight open-ended questions, six retrospective and two prospective or hypothetical. Quantitative responses were summarized descriptively, while qualitative responses were independently coded by two human researchers and two LLMs (GPT-4o and Claude 3.7 Sonnet) using a shared, human-developed codebook. Inter-rater reliability was assessed using Cohen’s and Fleiss’ kappa. Students reported high levels of engagement, motivation, perceived learning support, and perceived efficiency when using course-specific LLM applications; however, the study did not assess objective learning outcomes. Human-human agreement ranged from moderate to almost perfect (κ = 0.52–0.84), whereas average human-LLM agreement was lower and more variable (κ = 0.36–0.62). Claude aligned more closely with the human coders than GPT-4o across all eight questions when the two human-model kappas were averaged. Agreement was highest for concrete, task-focused prompts and lower for several reflective and prospective prompts, where the LLMs missed human-identified themes, fragmented broad themes into narrower subcodes, or assigned substantive meaning to non-responses. Hybrid intelligent feedback is used as a retrospective sensitizing framework rather than as a prospectively operationalized intervention. The findings support a bounded role for LLMs in qualitative educational research: models can assist with screening, discrepancy detection, and code suggestions, but human researchers remain necessary for contextual interpretation, validation, and adjudication.