‘ChatGPT did my homework’: lessons learnt from embedding a chatbot in undergraduate coursework

Abstract OpenAI’s ChatGPT, a formidable large language model (LLM) based on a generative pre-trained transformer architecture, has a remarkable, and rather unsettling, ability to solve conceptually challenging physics problems. One of the more impressive demonstrations of ChatGPT’s capabilities in this regard is its expert-physicist-level performance on the questions that make up the Force Concept Inventory [CG West, https://arxiv.org/abs/2303.01067 ]. Motivated by the deep implications of these advances in LLM technologies for student learning, attainment, and assessment, we asked a class of undergraduate physics students ( N = 119) to use ChatGPT to solve problems, and write Python code, related to the eigenfunctions of the Hamiltonian of the 1D quantum harmonic oscillator—a staple of introductory quantum mechanics courses. We discuss the significant pitfalls we encountered when incorporating ChatGPT into coursework in this manner, and make recommendations as to how LLMs might be better embedded into the undergraduate physics curriculum.