ARTIFICIAL INTELLIGENCE ASSESSING CONTENT-FOCUSED SHORT ANSWERS

The capabilities of Artificial Intelligence (AI), and specifically large language models (LLMs) have changed the way teachers work.Using LLM or other AI-assisted tools to help review student submissions has quickly become common practice.These AI-driven automatic assessment tools still have a lot of open questions regarding their effectiveness, performance, and reliability.In this study, we observe the LLMs' capabilities to assess textual answers.The data set used consisted of 31 different computer science-related questions and 2981 answers written in English with detailed feedback and the correct answers.The LLM we used was GPT-4o from OpenAI.At first, the performance of the LLM was tested against a single question present in the data set producing scores for all of its answers (N=82) using multiple different variations of settings.The best-performing approach was then used to process the full data set.With the full set, the model got the exactly correct evaluation in 41,3% of the cases.With an accepted error margin of 20%, the correctness was 74.7%.When observing the fully correct answers in the set (N=1802), the model was able to correctly identify 51.4% of them.The results can be used to guide future research endeavors in AI-driven automatic assessment research and to guide teachers on how to improve the performance of educational use of LLMs in different ways.