GenScore: agent-based short-answer question generation and scoring in software engineering courses

Short-answer questions are commonly used in educational assessments, as they are often viewed as a more effective way than multiple-choice questions to determine whether students have achieved the intended learning outcomes. However, manually creating appropriate questions targeting different cognitive levels such as those defined by the Bloom’s Taxonomy, and grading text answers from students are not trivial tasks for instructors. Existing work on auto-question generation and scoring in computing education typically targets coding-based questions. However, in software engineering courses, assessments can extend beyond coding to understanding of processes, DevOps methodologies, system design, etc. This work aims to address the dual problem of auto-generating and auto-scoring text based short-answer questions in software engineering courses. We propose an agent-based approach to question generation and scoring, in which each agent performs a distinct task and communicates with each other to complete its tasks. An agent can be realized using a selected large language model (LLM). To this end, we have implemented the proposed approach into a fully functional web app, utilizing OpenAI’s GPT-3.5 and GPT-4o models, which was deployed to support instructors and students in our course. We conducted a thorough evaluation of the agent-based approach using several public and private datasets. The results demonstrated the effectiveness of our approach. Among other insights, we observed that the LLM agents are good at generating questions at Bloom’s cognitive levels of Remember, Understand, Apply, and Analyze; and are good at scoring students’ answers when provided with 3 or more actual examples of human scoring.