IPRAG: Iterative Prompt Refining for Retrieval Augmented Generation

One of the inherent limitations of large language models (LLMs) is their propensity to generate hallucinated content, as the accuracy of generated texts cannot be solely guaranteed by the parametric knowledge they encapsulate. While retrieval-augmented generation (RAG) offers a practical complement to LLMs, its reliance on the relevance of retrieved documents raises concerns about how the model behaves when retrieval fails. In this paper, we propose a novel approach called Iterative Prompt Refining for Retrieval Augmented Generation (IPRAG) for enhancing the robustness of text generation. Our proposed approach comprises two key components: (1) a multi-dimensional evaluator designed to assess the overall quality of retrieved documents, and (2) an iteratively prompt refinement method that optimizes the generator’s prompts. In contrast to existing approaches, our IPRAG framework can be seamlessly integrated with various RAG-based methods, enabling a more comprehensive evaluation of their performance. The experiments conducted on public datasets covering short- and long-form generation tasks demonstrate that IPRAG can significantly enhance the performance of RAG-based approaches, thereby improving the quality and reliability of generated texts.