Comparison of Q&A Classification Using BiLSTM Based on Closed Domain Knowledge

The Faculty of Computer Science, Universitas Brawijaya (Filkom UB) is committed to providing quality services for the users especially internal and external stakeholders, one of which is through the HaloFilkom service. HaloFilkom services have limitations in terms of time. HaloFilkom services are not available 24 hours due to limited working hours. Questions asked by users are not answered directly. This weakness in the HaloFilkom system can be overcome by using a chatbot system. Chatbot is an interactive system that works with natural human language and can work 24 hours. Thus, the current study explores the basic chatbot model by classifying the Q&A in the closed domain knowledge. The dataset in this research is in the form of pairs of questions and answers regarding various topics at the Filkom UB. The knowledge is preprocessed using text preprocessing which includes case folding, tokenization, padding, and tensorization. One of the chatbot models is a generative model. Creating a generative chatbot model can be done using the Seq2Seq model mechanism which consists of an encoder and decoder. The model created consists of four different architectures, namely a model with an LSTM encoder without attention and with attention and a BiLSTM model encoder without attention and with attention. Hyperparameter tuning was conducted to obtain the best hyperparameter combination. The experiment results show the best hyperparameter combination obtained is hidden size 448, drop out rate 0.5, learning rate 0.001, batch size 64, and teacher force 0. The model with the best loss is obtained with a BiLSTM encoder architecture without an attention mechanism with a train loss of 0.120. The model with the highest BLEU Score was obtained by a model with a BiLSTM encoder architecture without an attention mechanism with a BLEU Score of 0.8587 on the training data. Testing using prompt testing obtained an average BLEU Score of 0.3745 on the BiLSTM encoder without an attention mechanism model. Finally, the best model architecture obtained in this research is BiLSTM encoder without an attention mechanism.