Evaluating AI dialogues in MOOCs: Learner perceptions of usefulness, cognitive engagement and usability

This paper describes a study that evaluated learner perceptions of artificial intelligence (AI)-supported dialogues integrated into six leadership-oriented massive open online courses (MOOCs). These structured interactions, delivered via Coursera Coach, were designed to prompt reflection, support personal insight, and reinforce course concepts. A mixed-methods approach was employed, combining survey data (n = 510) with thematic analysis of 285 open-ended responses. Three core dimensions were assessed: perceived usefulness, cognitive engagement, and usability. Results showed that 79 per cent of learners rated the dialogues as ‘very useful’, while 73 per cent found them ‘very easy to use’, indicating strong acceptance and accessibility. Cognitive engagement outcomes were also positive: 43 per cent reported that the dialogues challenged them to think in new ways, and 53 per cent stated they helped consolidate existing knowledge. Courses employing two AI dialogues (at the start and end) saw increased learner ratings in both usefulness and cognitive engagement over time, compared to single mid-course dialogues. Learner responses suggest that AI dialogues can offer a useful space for motivation, reflection, and applied thinking; however, feedback also identified limitations in naturalness of AI responses, onboarding clarity, and personalisation. These insights indicate that while conversational AI holds promise for supporting learner engagement at scale, its effectiveness depends on thoughtful implementation and continued refinement. This article is also included in The Business & Management Collection which can be accessed at https://hstalks.com/business/.