Exploring the Potential of Locally Run Large Language (AI) Models for Automated Grading in Introductory Computer Science Courses

This innovative practice full paper describes the effectiveness of self-hosted large language models (LLMs) in assisting with the automatic grading of CSI assignments. Educators often rely on automated review of student code submissions in larger courses. Despite recent advancements, current systems primarily focus on assessing functionality, with important aspects such as code structure, efficiency, and style often relegated to secondary foci. LLMs provide an increasingly attractive addition to these systems to enhance those overlooked areas. Prior research has shown LLM's capable of assisting students in understanding and resolving programmer error messages, correcting syntax errors, providing enhanced explanations of code segments, or even generating code. The absence of freely available, purpose-designed LLMs for grading and providing feedback on code submissions prevents widespread adoption by educators. Remotely-hosted systems, such as fine-tuned GPT models, have shown promise, yet the associated risks of privacy breaches, ethical considerations, and recurring costs make this approach unfeasible as a universal solution. To mitigate these concerns, self-hosted open-source models are an alternative that can operate on consumer-grade hardware and prevent some privacy and security concerns. While no purpose-built solution yet exists, it is unclear if any existing models are powerful enough to facilitate automated grading. To explore these questions, we present a two-phase analysis, leveraging real grading data from a semester length, introductory CSI course with 124 students and nine programming projects. Nine stable LLM models were selected and repeatedly prompted to grade student submissions using the same context that a human teaching assistant (TA) was given. This paper analyzes 1,172,383 API requests, totaling 33.4 days of active runtime, evaluating model consistency, ability to adhere to specified constraints, and comparison to human-generated grades. The results show various models' inability to consistently grade assignments, albeit with some exceptions. The importance of providing comprehensive context to models was highlighted, as incomplete contexts resulted in worse performance. Other models struggled with longer prompts, delivering less consistent results. Despite disparities between AI-generated and human-assigned grades, the potential for refinement is clear; improved rubrics or selective fine-tuning could enhance model output. Future work will focus on analyzing models' qualitative justifications for grades, refining rubrics, training on domain-specific datasets, and fine-tuning the highest performing models to potentially improve grading accuracy.