Performance Evaluation of a Chatbot with Retrieval Augmented Generation and Generative Pre-trained Transformer-4 Model for Taharah Domain

The chatbot is one of the developments of artificial intelligence. the chatbots are utilized as conversational agents employing large language models (LLMs). One challenge with LLMs is the occurrence of hallucinations in the generated responses. The author proposes a chatbot developed using retrieval-augmented generation (RAG) technique and the GPT-4 model. The chatbot retrieves additional information stored in a graph database as a source to generate responses to users. The chatbot is designed to answer questions regarding the foundational principles based on the four imams' interpretations found in the book Rahmah Al Ummah Fi Ikhtilaf Al A’immah. Based on testing results, the chatbot demonstrates accuracies, precisions, recalls, and F1-scores of $\mathbf{9 0 \%}, \mathbf{8 6 . 6 7 \%}, \mathbf{1 0 0 \%}$, and $\mathbf{9 2 . 8 6 \%}$, respectively.