Computer Practical Course Assessment with Large Language Models: A Case Study with ChatGPT

AI dominance is far and wide in education domain. It helps the teaching fraternity to automate most of their monotonic tasks. There is very serious research going on to find the reliability and capability of the LLM tools in assessing the students' exams. The evaluation process of the lab exams is a time consuming and sensitive task, particularly that involves a programming component in it. A team of researchers from University of Technology and Applied Sciences study the capabilities of both human and ChatGPT in evaluating the lab exam of Web Development course which involves HTML coding part in it. The team involved 22 students, who completed an 11-hour training in web development and conducted a lab exam for them. A clearly defined rubric was prepared for the evaluation purpose of this exam. Each student answer file was evaluated by 3 people, human, ChatGPT with Zero-Shot Learning (No rubrics) and ChatGPT with One-Shot Learning using a common rubric. Fleiss' Kappa calculation is used to measure the agreement among the evaluators. It is observed that the Fleiss' Kappa score is 0.506, indicating moderate agreement among Assessors. Still, this can be considered as a relatively high agreement among the raters, at the same time there is room for achieving higher agreement among the raters. Overall, the study concludes that when the assessment is based on the correct answer, the difference in evaluation is less compared to the partial and incorrect answers.