Can LLM Pass a CS1 Course?

ChatGPT and other Large Language Models (LLMs) have significantly impacted a number of fields by offering sophisticated natural language processing capabilities. LLMs produce remarkable outcomes in higher education, particularly in introductory computer science courses (CS1). This study examines the changing performance of two cutting-edge LLMs, ChatGPT and Gemini, in the setting of a university-level introductory computer science course (CS1). These LLMs were ‘enrolled’ as students for two consecutive academic years and given a wide range of assignments: lab exercises, midterm exam, and final exam. The fact that only one of the two LLMs finished the course in the first year showed the limitations of these models at that stage. But there was a noticeable change in the second year, when both LLMs received passing ratings, indicating the quick development of LLMs. This study highlights how LLMs can revolutionize computer science's higher education by creating chances for individualized learning, feedback and advanced teaching strategies. It also calls into question the authenticity of student work, academic integrity, and the necessity for teachers to modify their pedagogical approaches to successfully incorporate and utilize these powerful technologies while upholding the fundamental principles of their courses.