Can Language Models Grade Algebra Worked Solutions? Evaluating LLM-Based Autograders Against Human Grading

In this study, we investigate the feasibility of using LLMs as autograders by evaluating multiple models—OpenAI's GPT-4o, GPT-4.1-mini, and GPT-4.1-nano—on a manually graded open-source dataset of college algebra worked solution responses. For each model, we assess performance both with and without a self-consistency grading approach. We compare LLM-generated correctness labels to human annotations across 18,000 responses. Results show that GPT-4.1-mini achieves the highest accuracy (94.47\% with self-consistency), followed by GPT-4.1-nano (93.07\%) and GPT-4o (91.93\%). These findings suggest that self-consistency can slightly improve grading reliability, and that even compact models like GPT-4.1-mini and GPT-4.1-nano can approach human-level agreement in algebra autograding.