On the Effectiveness of the ChatGPT Model with Fine-tuning in Question-Answer Systems on Slack

QABot is an app on Slack designed for universities. When students ask questions on the app, they receive responses from teachers and TAs, but the system is designed so that students do not know who has responded. In other words, all messages, such as questions and answers, between students and teachers are forwarded to the local server where the app is running. By using the app, students no longer directly ask specific teachers or TAs via DM, and the workload can be distributed. However, there are cases where similar questions arise from various students. In this paper, we have given QABot a mechanism for creating answer drafts using ChatGPT. Three models of ChatGPT are examined: 4o mini, 4o mini using fine-tuning, and 4o mini using Simple RAG, which can refer to past questions and answers. These models are evaluated by using BertScore. 4o mini using fine-tuning demonstrated good results with statistical significance.