An Empirical Study of the Error Characteristics in an Online Judge System

An online judge system (OJS) is a system that automatically compiles, verifies, and executes programs submitted by users for specific problems, and provide feedback. OJSs offer a vast array of programming-related problems, enabling users to choose relevant problems and engage in self-directed programming and algorithms learning. However, OJS research and learning support using OJS remain limited and insufficient. We analyzed the submission histories of AtCoder, an OJS platform included in CodeNet, focusing on error resolution rates, the time required for resolution, and the variation of these metrics based on the experience of users in solving problems of varying difficulty levels. We made the following findings: (1) Time limit exceeded error is the most difficult to solve regardless of the programming proficiency of users. (2) Reversal in the proportions of compilation errors and time limit exceeded errors before and after problems of specific difficulty level. (3) Error resolution time decreases with the programming proficiency of users regardless of error types. These findings are crucial in identifying errors in which users require support and enabling competition organizers to assess the programming abilities of participants.