GenAI and the Mirage of Personalised Learning for All

Generative artificial intelligence (GenAI) is gaining attention in education, particularly for its potential to personalise learning. However, studies have yet to assess its effectiveness and identify its limitations within controlled educational contexts across various disciplines and GenAI models. This study aims to help fill this gap by evaluating ‘SmartTest’, a GenAI chatbot designed by the authors to prompt questions, offer immediate feedback, and stimulate critical thinking through conversational nudges. Over five test cycles, SmartTest was used in a criminal law course at an Australian university. The results revealed SmartTest struggled with complex structured problem-solving exercises. While SmartTest showed some promise in aiding short-answer question learning, it remained limited and prone to inaccuracies. These findings highlight concerns that inflated expectations of GenAI could mislead educators and investors about its capabilities to deliver economically viable and quality personalised learning.