A Comparative Evaluation of Retrieval-Augmented Generation For Arabic Documents

Generative AI creates new content using AI models (i.e., LLMs) trained on large datasets. However, LLMs lack specific sources and can produce inaccurate or misleading information. Retrieval-augmented generation (RAG) improves accuracy by combining LLMs with a content store, providing evidence for responses and making the model more adaptable to changing information. This paper introduces a comparative evaluation of various generative AI models—LLama 3, LLama 3.1, Gemma, Gemma 2, Phi 3, Phi 3.5, and Phi 3.5 Mini based on their performance in answering questions derived from Arabic and English documents using the RAG locally. The models were evaluated based on their ability to answer questions of varying difficulty levels. Questions were categorized as easy, moderate, or complex. The models’ performance was measured in terms of two key metrics: response time to answer each question and the accuracy of their responses (or quality of the answer). Using RAG method, the results show that LLaMA 3.1 consistently outperformed other models in both Arabic and English languages, whereas Gemma 2 demonstrated decent results. Additionally, response time can be misleading indicators of model performance because less accurate responses require less effort and are generated more rapidly than highly accurate ones.