Assessing the Consistency of Open-Source Large Language Models for Algorithm Evaluation

The grading of open-ended questions in education is labor-intensive and subject to human error, making it an attractive target for automation through artificial intelligence. This work explores the grading consistency of four open-source large language models (LLMs) in rubric grading algorithms on design, completeness, clarity & readability, and logic. Statistical methods revealed that Anthropic Claude was the most consistent scorer, with an average normalized standard deviation (SD) of 0.17 points and an intraclass correlation coefficient of 0.828, while Microsoft Copilot was the least consistent. The “completeness” rubric category had the lowest average SD, indicating it was graded the most consistently.