You are the auditor: A ChatGPT-based multiple choice exam

This case study examines the use of ChatGPT, a chatbot that utilises generative artificial intelligence (GenAI) technology, to amplify the efficacy of multiple-choice examinations in a financial accounting course. The primary objective is to introduce students to the advantages and limitations of GenAI tools such as ChatGPT while simultaneously fostering the development of critical thinking skills in their application. By demonstrating the tool’s ability to give both correct and incorrect responses, students will deepen their understanding of its appropriate use and develop a professional scepticism towards AI-generated responses, which can ‘hallucinate’ and yield incorrect answers to AI prompts. Furthermore, this study seeks to explore the role of AI-supported tests in fostering students’ problem-solving abilities and analytical acumen. The research examines student performance by giving exam questions, an AI-generated response and additional alternative answers in a multiple-choice format. On average, ChatGPT 3.5 responds with correct answers to accounting questions about 50 per cent of the time. To succeed in such examinations, students must evaluate if the ChatGPT answer is correct or if another of the given answers is more accurate. The case study also highlights the advantages and challenges of incorporating AI into accounting pedagogy, addressing concerns about the assimilation of this advanced technology.