Investigating Autograder Usage in the Post- Pandemic and LLM Era

This work investigates the impact of Large Language Models (LLMs) and the COVID-19 pandemic on student behavior with autograder systems in three programming-heavy courses. We examine whether the release of LLMs like ChatGPT and GitHub Copilot, along with post-pandemic effects, has modified student interactions with autograders. Using data from student submissions over five years, totalling over 4,500 students across over 420,000 submissions, we analyze trends in submission behaviors before and after these events. Our methodology involves tracking submission patterns, focusing on timing, frequency, and score.