The Role of AI Chatbots in Engineering Education: Experimental Findings and Implementation Strategies

In the field of education, the recent revolution in the large language model (LLM) space has enabled a whole host of interesting applications, such as content generation, support, and even personalized learning. While there are many ad-hoc experiments in flight, scientific studies on the effectiveness of these techniques have been limited. In order to increase the scientific rigor and potential for experimental reproducibility, the Tallinn University of Technology (TalTech) team deployed an artificial intelligence (AI) chatbot within the context of a traditional mainstream mechanics physics course and instrumented the class to facilitate a scientific study on utility. The AI chatbot focused on course support and tutoring in the Estonian language, and the scientific design-for-experiment focused on impact for students, instructors, and course designers. The study revealed measurable gains in instructor productivity and student access. The study also demonstrated the expected need for additional due diligence required to manage AI hallucinations. Perhaps most interestingly, the study revealed the unexpected benefits of cataloguing student chat interactions as a rich data source for the development of instructional materials and future course design. In fact, LLMs were also very useful to evaluate these AI chatbot conversations. Overall, this scientific study provides insights for the educational community into the leverage of using AI chatbots for instruction and in dramatically increasing access by enabling the use of a local language.