Aftina: enhancing stability and preventing hallucination in AI-based Islamic fatwa generation using LLMs and RAG

Abstract Question–answering (QA) systems face considerable challenges when involved in Islamic fatwas due to the complexity and sensitivity of the data. Such problems involve providing accurate and reliable responses, managing hallucinations and inaccurate responses, and maintaining the stability of the generated responses. Prior studies have concentrated mainly on collecting and preprocessing Islamic datasets or developing retrieval-based QA systems, overlooking the precision and reliability required for fatwa issuance. To address this issue, we propose a QA approach utilizing advanced retrieval-augmented generation (RAG), which is enhanced by a re-ranker to increase response stability, eliminate hallucinations, and prioritize the most appropriate and exact answer. This enhancement significantly improves response stability and reduces hallucinations by improving the data used for answer generation. We conducted experiments across three setups: (1) base LLM, (2) LLM with RAG, and (3) LLM with RAG and re-ranker. The third method of LLM with RAG includes a re-ranker for knowledge retrieval, which improves the process and ensures relevant and trustworthy data. This differentiates it from the second method, which uses a retrieval model. The Flash re-ranker retrieves the most relevant data, which increases the response stability and trustworthiness. Evaluations using BERTScore, hallucination, completeness, and irrelevance metrics demonstrated that the third experiment LLM with RAG and re-ranker outperformed other setups, providing precise, stable, and dependable answers. This research contributes a robust methodology to improve AI-driven fatwa systems, guaranteeing higher precision and trustworthiness in Islamic QA systems.