Case Study of Improving Educational Chatbots with Customized Information Retrieval

Generic chatbots can function as virtual tutors, assessing responses and providing feedback. However, trained on generalized data limits them to act as subject-specialist tutors, resulting in weak or inaccurate responses and feedback, particularly for complex queries. We aimed to develop and evaluate whether a specialised information retrieval system can enhance a generic chatbot’s capabilities. The customized chatbot was integrated into a teaching session and evaluated by the students. Our approach enabled the chatbot to access relevant information and provide tailored feedback, resulting in 75% of students finding the teaching session engaging and helpful. Students also expressed a preference for the customized chatbot over a generic one. However, occasional inaccuracies in responses to highly complex queries occurred and were attributed to incomplete optimization of the retrieval system. To address this, we recommend developing a comprehensive knowledge database and fully optimising the retrieval system to ensure consistently accurate and contextually relevant responses.