Empowering Educators: Towards a GPT-Based Approach to Automate Unit Test Generation

Assessing code automatically is a significant challenge in distance learning, especially in large online courses with limited teaching resources. Although auto-gradable programming exercises address scalability, creating enough high-quality exer-cises-particularly designing comprehensive unit tests-remains time-consuming and labor-intensive. To address this, we introduce a GPT-based feature that automates unit test generation for customized exercises. With a single button press, instructors can adapt existing exercises to meet specific teaching objectives while preserving auto-gradability. The AI-generated tests comprehensively cover potential edge cases that might otherwise be overlooked, thus reducing the need for manual oversight. An empirical evaluation with eight experienced educators showed these tests to be both thorough and time-efficient, achieving an average System Usability Scale (SUS) score of 81.79. Participants, who reported intermediate to advanced proficiency in designing manual unit tests and intermediate familiarity with AI tools like ChatGPT, praised the feature's ease of use and seamless workflow integration. Their combined expertise in teaching, coding, and AI-informed course development allowed them to provide insightful feedback on the practicality and reliability of our GPT-based solution. Our study includes a small participant pool (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathrm{n}=8$</tex>) and primarily focuses on Python, a language wellsupported by GPT. Future research will involve expanding the participant group, exploring additional programming languages, and assessing long-term tool performance and adaptability in diverse educational contexts. By harnessing GPT's language modeling capabilities, our approach addresses the gap between generic, limited-coverage test generation and the need for robust, domain-specific tests. Early reports from participants suggest that specialized exercises-such as those involving advanced data structures-can also benefit from automated unit test generation, though further evaluation is necessary. By leveraging artificial intelligence, this method streamlines exercise customization and enhances the overall usability and effectiveness of programming education tools. It has the potential to revolutionize auto-gradable exercise creation at scale, empowering educators to deliver high-quality instruction while tackling both the technical and pedagogical challenges in programming education.