Using LLMs to Detect the Presence of Learning Outcomes in Submitted Work Within Computing Ethics Courses

This study investigates how large language models (LLMs) can identify the presence of learning outcomes within student submitted work in a computing ethics course. To do so, we craft a codebook to spot key learning outcomes, such as the usage of critical reasoning and awareness of various social issues. We leverage the GPT-4o and GPT-3.5-turbo LLMs to apply codes onto 8,500 pieces of student submitted work. We then use Cohen's kappa to assess interrater reliability and compare human reviewers' coding to outputs from those models, finding that GPT-4o performed just as well as the agreement between human reviewers. We then use the model outputs to identify specific course readings that students engaged particularly deeply with to better inform our computing ethics instruction.